Best skill-generation pipeline clears just 14% of code-repo tasks, 25% of document tasks

Notes on SkillGenBench: Benchmarking Skill Generation Pipelines for LLM Agents (arXiv:2605.18693) — Yifan Zhou, Zhentao Zhang, Ziming Cheng, Shuoming Zhang, Qizhen Lan, Zhangquan Chen, Zhi Yang, QianyuXu, Ronghao Chen, Huacan Wang, Sen Hu · May 2026

Note published July 25, 2026 · written by SkillFed’s research pipeline from the paper above · how these notes are made

AI-assisted notes · reviewed by SkillFed Agentic benchmarks Bridge: benchmarks × evolution

SkillGenBench splits the skill lifecycle in two and scores only the first half: not whether an agent can use a skill, but whether a pipeline can generate one. A generator sees only raw source material — a pinned code-repository snapshot or a long-form document bundle — and has to produce a standalone SKILL.md package. A separate, fixed executor (MiniMax-2.5) then runs that package against hidden, deterministic test cases, so generation quality is measured apart from prompting skill or task planning. The benchmark crosses two source types — repository-grounded, where the procedure is scattered across code, config, and scripts, and document-grounded, where it's explicit but spread across a manual or spec — with two regimes: task-conditioned generation, where the task is revealed before the skill is written, and task-agnostic generation, where a reusable skill library has to be distilled before any task is known. The 187 accepted tasks went through a five-stage pipeline — knowledge-graph extraction, scenario generation, test-case generation, a difficulty filter that throws out anything solvable without the corpus or trivially solvable with it, and reference-skill verification — plus human review. Five generation methods, from naive prompting to workflow-based and self-evolving pipelines, each paired with six backbone models — Claude Sonnet 4.5, GPT-5, Kimi K2.5, GLM-5, MiniMax-M2.7, and Qwen3.6-Plus — and scored on pass@3.

Even the best combination is far from solved. Averaged across all six backbones, the strongest method (SkillSeekers) reaches only 14.4% pass@3 on repository tasks and 25.0% on document tasks — every other method scores lower, and several fall below the no-skill baseline entirely. Repository tasks are uniformly harder than document tasks (10.8%–14.4% versus 21.4%–25.0% across methods): recovering implicit execution structure — environment setup, command conventions, data flow — from code is a different problem than reading constraints off a page. Structural quality and execution success don't track each other, either: SkillNet scores best on environment and grounding, SkillCreator on contract and procedure, but SkillSeekers — weakest on most of those structural axes — wins on actual pass rate. A skill package that looks complete is not the same as one that runs. The failure signature shifts with the source: code-repo failures skew toward runtime or dependency breakage (53%), code-documentation failures skew toward interface or schema mismatches (85%), and domain-knowledge-document failures split between state or rule errors (44%) and numeric or formula errors (37%). Task-agnostic generation compounds all of this — without a task to aim at, generators produce libraries that sometimes underperform having no skill at all — and more generation tokens only help up to roughly 24,000–64,000 tokens before gains flatten out.

Key numbers

Benchmark size	187 tasks (repo- and document-grounded)
Best pass@3, code-repo tasks	14.4% (avg. across 6 backbones)
Best pass@3, document tasks	25.0% (avg. across 6 backbones)
Code-repo failure cause	53% runtime/dependency errors
Generation-budget plateau	gains flatten past ~24K–64K tokens

What this means for skill authors

For anyone building an agent skill from a codebase or a spec sheet rather than downloading one, this is the sobering baseline. The best current pipeline, tuned across six frontier and near-frontier backbones, still fails on the majority of repository tasks and roughly three-quarters of document tasks. Auto-generating a working skill from a README and a directory tree isn't a solved problem yet — it's closer to a research-grade open one. That's exactly the gap a vetted catalog of already-verified skills is built to close, rather than leaving generation to run at inference time.

The static-versus-dynamic mismatch is the sharper lesson here. A skill artifact can pass every structural check — clear contract, documented environment, explicit constraints — and still not execute, while a structurally weaker artifact wins on the only metric that matters: does the task actually pass. Field-presence linting of a SKILL.md isn't verification; this paper shows it can point the wrong direction entirely. That's the case for checking what a skill does when run, not just what it claims to do — the same reasoning behind treating execution checks as a gate rather than a formatting pass.

The source-specific failure modes double as a map of where to look first. Code-repository skills tend to break on runtime and dependency issues. Skills built from API docs break more on interface and schema mismatches. A skill distilled from a rules-heavy manual is more likely to trip on state or numeric errors. None of that is exotic — it's the ordinary failure surface of translating prose or code into something callable — but it means the failure category tells you more than a single pass-rate number does.

Skills related to this research

lwy-project-skill-writer This skill transforms recurring manual tasks into standardized, reusable AI capabilities by understanding your problem, analyzing your project structure, and generating a complete skill design. It guides you through problem classification, project profiling, and skill architecture before generating executable SKILL.md files that integrate seamlessly with your codebase.★ 2 compound-learnings Extract patterns from recent sessions and convert them into persistent capabilities—rules for heuristics, skills for workflows, and hooks for automation. The skill guides you through gathering learnings, consolidating similar patterns, detecting meta-themes, and proposing artifacts with appropriate signal thresholds before creation.★ 3,875

Related notes

References

Zhou, Zhang, Cheng et al., 2026. SkillGenBench: Benchmarking Skill Generation Pipelines for LLM Agents. arXiv:2605.18693.
Li et al., 2026. SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks. arXiv:2602.12670.
Han et al., 2026. SWE-Skills-Bench: Do Agent Skills Actually Help in Real-World Software Engineering? arXiv:2603.15401.
Alzubi et al., 2026. EvoSkill: Automated Skill Discovery for Multi-Agent Systems. arXiv:2603.02766.
Liang et al., 2026. SkillNet: Create, Evaluate, and Connect AI Skills. arXiv:2603.04448.