39 of 49 Real-World SWE Skills Move the Pass Rate by Exactly Zero

Notes on SWE-Skills-Bench: Do Agent Skills Actually Help in Real-World Software Engineering? (arXiv:2603.15401) — Tingxu Han, Y. Zhang, Wei Song, Chunrong Fang, Zhenyu Chen, Youcheng Sun, Lijie Hu · March 2026

Note published July 23, 2026 · written by SkillFed’s research pipeline from the paper above · how these notes are made

AI-assisted notes · reviewed by SkillFed Agentic benchmarks

SWE-Skills-Bench tests whether packaging procedural knowledge as an agent skill changes outcomes on real software engineering work, not on synthetic coding puzzles. It starts from a pool of over 84,000 public skills and filters down to 49 spanning six SWE subdomains — deployment and DevOps, analytics and monitoring, API development, data science and ML, security and testing, and developer tools — each paired with an authentic GitHub repo pinned at a fixed commit plus a requirement document carrying explicit acceptance criteria. A requirement-driven verification framework turns each acceptance criterion into an executable pytest check, so every task instance (about 565 total, roughly 11 per skill) gets a deterministic pass/fail verdict instead of a judged score. Every task runs twice — once with the skill's SKILL.md dropped into the agent's context, once without — under a single fixed agent configuration (Claude Code driving Claude Haiku 4.5), so the skill's presence is the only thing that changes.

The paired design exposes a benefit far thinner than skill-adoption enthusiasm suggests: 39 of the 49 skills move the pass rate by exactly zero, and the average gain across all 49 sits at +1.2%. Token usage isn't coupled to that outcome at all — among skills that already hit a 100% pass rate with or without help, some cut token spend while others drove it up by as much as 451%, so a skill can visibly change how the agent reasons without changing whether it succeeds. Only seven skills earned their keep, with gains up to +30% concentrated where the skill supplied something the model genuinely lacked — a financial risk-metrics formula, a specific CI pipeline pattern, a traffic-management convention — sometimes while cutting token cost too. Three skills made things worse, by as much as −10%, through what the paper calls context interference: a hard-coded, version-specific template anchors the agent on details that conflict with the actual target repo, occasionally pushing it to fabricate fields to paper over the mismatch.

Key numbers

Skills with zero pass-rate gain	39 of 49
Average pass-rate gain, all 49 skills	+1.2%
Skills with meaningful gain (max)	7 skills, up to +30%
Skills that degraded performance (max)	3 skills, up to −10%
Token overhead despite unchanged pass rate	up to +451%

What this means for skill authors

The uncomfortable number for skill authors is the 39, not the 1.2%. A skill that produces zero measured lift usually means one of two things: the base model already knew the workflow, or the skill described it at the wrong altitude — too generic to add information, or too narrowly tied to the one repo it was written against. The gains that did land were technical and specific: a formula, a pipeline convention, a protocol detail the model couldn't reconstruct on its own. That's the shape of a skill worth publishing, not a restatement of something the model already knows.

The token-overhead decoupling is the sharper warning for anyone judging a skill by a demo. A skill that makes the agent write and reason more looks active and thorough, right up until you check whether the extra tokens bought a passing test — here, mostly, they didn't. That's the exact failure mode a vetted catalog and dual-tool scanning exist to catch: not whether a skill runs, but whether running it, with the task held constant, actually moves the acceptance criteria — the same paired with/without comparison this benchmark runs at scale.

The three degraders matter for how a catalog ages a skill, not just how it admits one. Each failure traced back to a version-specific template colliding with the target project's actual versions — a skill written correctly for one API generation actively misleading an agent working against another. Vetting a skill once at submission time doesn't catch that; it only shows up once the skill's assumptions about dependency versions drift out from under the projects it gets used on.

Skills related to this research

Openscad Collision Detection This skill helps you find and analyze collisions between 3D objects within OpenSCAD designs. By detecting overlapping geometry and intersection points, you can validate your models and catch design conflicts before fabrication. Perfect for engineers and makers who need to verify that components fit together correctly.★ 1 Network Doctor Network Doctor guides you through systematic network diagnostics without risky guesswork. It establishes baseline measurements first, isolates root causes across ISP, Wi-Fi, DNS, MTU, proxy, and background processes, then proposes only reversible fixes with before-and-after verification.★ 5

Related notes

References

Han, T., Zhang, Y., Song, W., Fang, C., Chen, Z., Sun, Y., & Hu, L. (2026). SWE-Skills-Bench: Do Agent Skills Actually Help in Real-World Software Engineering? arXiv:2603.15401.
Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., & Narasimhan, K. (2024). SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
Chen, M., et al. (2021). Evaluating Large Language Models Trained on Code (HumanEval).