Skill pass rates jump from 33.6% to 88.0% — with zero gradient steps

Notes on Skills-Coach: A Self-Evolving Skill Optimizer via Training-Free GRPO (arXiv:2604.27488) — Yu Tian, Jiawei Chen, Lifang Zheng, Ming Tao, Xinyi Zeng, Zhaoxia Yin, Hang Su, Xian Sun · April 2026

Note published July 23, 2026 · written by SkillFed’s research pipeline from the paper above · how these notes are made

AI-assisted notes · reviewed by SkillFed Skill evolution

Skills-Coach puts a skill through a four-stage self-improvement loop. First, a task generator reads the skill's own specification and writes a test suite of standard, advanced, and boundary-case tasks. Then a Training-Free GRPO optimizer rewrites the skill's instructions and code in parallel — generating multiple variants and scoring them against each other, with no gradient updates and no backprop through model weights. A Comparative Execution Module runs the original and the optimized version on identical tasks inside isolated environments; a Traceable Evaluation Module then grades both against 51 explicit criteria spread across 8 dimensions, with the pass line set at 70% of the maximum score. Execution comes in two modes — a fast 'virtual' mode that estimates completion from keyword matches and hash-seeded pseudo-randomness, and a 'real' mode that actually runs the skill and inspects its output files, logs, and errors. To test the whole pipeline, the authors built Skill-X, a 48-skill benchmark drawn from the Anthropic, Clawhub, and Vercel Labs skill libraries — 29 instruction-only, 19 shipped with code.

Across all 48 skills, average task score rose from 0.37 to 0.84 and pass rate climbed from 33.6% to 88.0%, using just 3 optimization epochs and 12 generated tasks per skill. Code-bearing skills started furthest behind — a 26.97% pass rate — but gained the most in relative terms, closing to 82.89%; instruction-only skills moved from 37.93% to 91.38%. Twenty-three of the 48 skills gained at least 0.5 in score. Four went from a dead 0.0 to a perfect 1.0, Browser and MCP-Builder among them, and only one skill improved by less than 0.1 — a single near-miss inside an otherwise near-universal lift.

Key numbers

Pass rate across 48 skills	33.6% → 88.0%
Average task score, relative gain	0.37 → 0.84 (+127%)
Code-inclusive skills, starting pass rate	26.97%
Skill-X benchmark size	48 skills (29 instruction-only / 19 code)
Training tasks used per skill	12 (3 epochs, 8 held-out test tasks)

What this means for skill authors

The real finding here is that skill quality is cheap to fix once it's measurable. Skills-Coach doesn't need a reward model or a labeled dataset of thousands of trajectories — it needs a way to generate test tasks from the skill's own spec and a way to score the result. From there, an LLM iterates on its own instructions and code like an unusually literal editor. For anyone authoring a skill for a team library or a marketplace, the practical read is blunt: an unoptimized skill.md is plausibly leaving 40-50+ points of pass rate on the table, and closing that gap here took generated test cases and a few rewrite epochs, not a training run.

The split between code-bearing and instruction-only skills matters more for catalog curators than for solo authors. Code-inclusive skills started lowest and improved the most in relative terms — read that as: code paths fail for mechanically fixable reasons (missing input validation, uncached calls, unhandled errors) more often than prose paths do. That's exactly the class of defect worth catching before a skill reaches a catalog rather than after an agent hits it live — the same logic behind screening skills with more than one independent tool instead of trusting a single pass.

One caveat worth carrying forward: the paper's 'virtual mode' scores completion from keyword presence and a hashed pseudo-random check, not real execution, so its cheaper numbers are proxies. The real-mode results are the ones that actually tell you whether a given optimized skill is production-ready.

Skills related to this research

Deep Learning Master production-grade deep learning using PyTorch, covering neural architectures, transformer models, and optimization techniques. Learn training best practices including mixed precision, gradient management, and deployment strategies for real-world applications.★ 4 idea-generator Idea Generator walks founders through structured discovery to uncover product concepts grounded in what they already know or do. It captures context through targeted questions, synthesizes patterns into candidate directions, scores each against five key axes, and writes a shareable product-idea.md document that feeds into downstream planning skills.★ 168 codspeed-optimize This skill acts as an autonomous performance engineer, using CodSpeed benchmarks and flamegraph analysis to iteratively optimize code. It measures baseline performance, identifies bottlenecks through flame graphs, applies targeted changes, and validates improvements across simulation and walltime modes. The skill handles the full optimization loop—from establishing baselines to comparing runs and reporting gains—stopping only when significant improvements plateau or the user decides to halt.★ 235 codspeed-setup-harness This skill guides you through configuring benchmarks and integrating CodSpeed into your project. It covers language-specific harnesses (divan, criterion, pytest-benchmark, vitest, go test, Google Benchmark) and a universal exec mode for any language, helping you identify performance-critical code and establish repeatable performance tracking.★ 235

Related notes

References

Tian, Y., Chen, J., Zheng, L., Tao, M., Zeng, X., Yin, Z., Su, H., & Sun, X. Skills-Coach: A Self-Evolving Skill Optimizer via Training-Free GRPO. arXiv:2604.27488 (2026).