Skills Pass the Task While Skipping 54-61% of Their Own Instructions

Notes on Skill Coverage: A Test Adequacy Metric for Agent Skills (arXiv:2606.20659) — Boyin Tan, Xiaowei Huang, Youcheng Sun · June 2026

Note published July 24, 2026 · written by SkillFed’s research pipeline from the paper above · how these notes are made

Task success hides which parts of a skill an agent actually used. Skill coverage is a new test adequacy metric built to expose that: it parses each skill's natural-language instructions into semi-structured Skill Behavior Constraints (SBCs) — condition/behavior pairs in an EARS-style "when X, the agent shall Y" form — then checks every agent trajectory against every constraint, labeling each one "not covered," "pass," or "fail" from observable evidence: messages, tool calls, file state. Applied to SkillsBench, the framework extracted roughly 4,283 constraints across 202 task-linked skills. An LLM judge (DeepSeek V4 Flash) had to earn its keep against human annotators before being trusted at scale: 88.58% agreement on whether a constraint was covered at all, 81.89% on the full three-way label, against a human-human ceiling of 98.05% (κ=0.961) — proof the labeling rubric itself is reproducible, not just the judge.

Across five agent-model configurations on the SkillsBench leaderboard, trajectories covered only 38.66% to 45.51% of the extracted constraints on average — most of a skill's documented behavior goes unexercised even in runs that succeed. Successful tasks covered somewhat more of the skill than failed ones and passed slightly more of what they did cover (73.3% vs. 69.5% pass rate), but the gap is narrow enough to make the point stick: task success and skill coverage measure different things, and a passing trajectory can still ignore most of a skill's instructions. Using Fail verdicts as a map, skills got rewritten to re-emphasize only the instructions agents had actually failed to follow — no new content, just sharper emphasis — and rerunning the same tasks recovered 16.0% of previously failed tasks on average across the five configurations.