Curated Skills Add 16.6 Points — Self-Authored Ones Cost You 11.5

Notes on SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks (arXiv:2602.12670) — Xiangyi Li, Wenbo Chen, Yimin Liu, Shenghan Zheng, Xiaokun Chen, Yifeng He, Yubo Li, B. You, Haotian Shen, Jiankai Sun, Shuyi Wang, Qunhong Zeng, and 30 more · February 2026

Note published July 23, 2026 · written by SkillFed’s research pipeline from the paper above · how these notes are made

AI-assisted notes · reviewed by SkillFed Agentic benchmarks

SkillsBench runs Agent Skills — structured procedural packages loaded into an LLM agent's context at inference time — through a controlled, paired test instead of a single-condition demo. The benchmark assembles 87 tasks across 8 domains (software engineering, natural science, cybersecurity, finance, and more), each shipped with a curated Skill and a deterministic verifier so pass/fail isn't a judgment call. The task pool itself passed through selection: 400 candidate submissions from 142 contributors, an automated gate (structural integrity, oracle execution, AI-text detection, leakage checks), then human review of at least 30 minutes per task, for a 22% acceptance rate. Every task runs three trials under matched no-Skills and curated-Skills conditions, across 18 model-harness combinations spanning OpenHands, Claude Code, Gemini CLI, and Codex CLI paired with models like GPT-5.5, Gemini 3.1 Pro, GLM 5.1, and MiniMax M2.7.

Average pass rate across all 18 configurations climbs from 33.9% without Skills to 50.5% with them — a 16.6 percentage-point gain, or a 25.5% normalized gain against the no-Skills ceiling. The gain is uneven: individual configurations range from +4.1 to +25.7 points, and domains split just as widely, from +28.8 points in natural science down to +11.6 in software engineering. Bundle size matters more than raw coverage — a single focused Skill adds +18.0 points, two-to-three add +19.0, and four-or-more collapse to +10.1, roughly half the benefit. A smaller model carrying Skills (MiniMax M2.7 at 34.9%) outperforms a larger model running bare (GLM 5.1 in OpenHands without Skills, at 32.7%). The sharpest result: Skills a model writes for itself don't just underperform curated ones, they push pass rates 8.1 to 11.5 points below the no-Skill baseline on the same configurations where curated Skills add 18.2 to 24.8 points.

Key numbers

Curated Skills avg pass-rate gain	+16.6 pp (33.9% → 50.5%)
Normalized gain	25.5%
4+ skill bundles gain (vs +19.0 pp for 2-3)	+10.1 pp
Self-authored Skills vs no-Skill baseline	-8.1 to -11.5 pp
Tasks with negative Skill delta	13 of 87

What this means for skill authors

For skill authors, the bundle-size result is close to a design rule: pack no more than three modules into a Skill before the marginal instruction starts subtracting value instead of adding it. The failure analysis backs this up — of the 13 tasks out of 87 where Skills made things worse, the recurring causes were heavyweight pipelines that displaced a model's already-strong default strategy, not missing capability. Comprehensiveness reads as diligence to a human reviewer; to the agent consuming it at inference time, it's often just noise competing with a plan it already had working.

The more consequential number is what happens when a model writes its own Skill: 8.1 to 11.5 points below doing nothing at all, on the exact configurations where a curated version of the same Skill adds 18.2 to 24.8 points. That gap isn't a marginal edge case — it's the difference between a Skill helping and a Skill actively sabotaging the run, and it makes the strongest case yet for treating a skill catalog as a vetting problem rather than a hosting problem. Unreviewed, self-published instructions aren't neutral filler; they're a measurable regression risk, which is the qualitative argument for scanning and curating skills before they ever reach an agent's context.

There's a portability wrinkle too: the same model shows different gains depending on which harness surrounds it (Gemini 3.1 Pro reads 60.8% natively but drops to 52.8% inside OpenHands), so a Skill validated in one agent runtime doesn't automatically carry its measured benefit into another. Authors and evaluators should treat harness as a variable in any efficacy claim, not a footnote to it.

Skills related to this research

Rtk Rtk helps you set up and troubleshoot Rust Token Killer, a CLI tool that rewrites shell commands into token-efficient versions for AI agents. Install via Homebrew, script, or Cargo, then initialize RTK for your specific agent—Claude Code, Codex, Gemini CLI, Cursor, Copilot, Windsurf, Cline, or OpenCode. Use direct `rtk` commands like `rtk git status` or `rtk read` when automatic hooks aren't available.★ 38 codspeed-setup-harness This skill guides you through configuring benchmarks and integrating CodSpeed into your project. It covers language-specific harnesses (divan, criterion, pytest-benchmark, vitest, go test, Google Benchmark) and a universal exec mode for any language, helping you identify performance-critical code and establish repeatable performance tracking.★ 235

Related notes

References

Li, Liu, Chen, et al., "SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks" (arXiv:2602.12670, 2026)