LLM-Written Agent Skills Add Zero Pass-Rate Gain — Self-Diagnosis Closes Up to 67% of the Gap to Human Authors

Notes on SkillAxe: Sharpening LLM-Authored Agent Skills Through Evaluation-Guided Self-Refinement (arXiv:2606.10546) — Srishti Gautam, Arjun Radhakrishna, Sumit Gulwani · June 2026

Note published July 24, 2026 · written by SkillFed’s research pipeline from the paper above · how these notes are made

SkillAxe treats skill quality as a diagnosis problem, not a rewriting problem. It runs each task twice — once with a skill injected, once without — and scores the gap along four axes. An LLM judge calls the direction and magnitude of the quality shift. Trigger precision checks how cleanly a skill's description separates tasks that should activate it from ones that shouldn't, measured as embedding-space distance between positive and negative trigger phrases. Instruction compliance gets a fault attribution pass that splits a broken rule into two diagnoses: the agent ignored good guidance, or the guidance itself was bad. And solution-path coverage asks how much of the space of valid approaches a skill's content actually supports. None of this needs ground-truth labels, hand-written test suites, or environment reward signals — the improvement brief comes entirely from comparing agent behavior with and without the skill.

On SkillsBench's 89 tasks, human-written skills lift pass rate by 16.2 percentage points; skills an LLM writes for itself add no measurable gain. One round of SkillAxe self-refinement turns that zero into a 28% relative improvement and closes 47–67% of the remaining gap to human authors. The mechanism is coverage, not correctness: refined skills get agents to produce a gradeable output on 72.7% of tasks versus 46.7% unaided, while answer quality among tasks that do complete is unchanged either way (57.1%). Deployed as a running improvement loop on SpreadsheetBench instead of a one-shot eval, the same diagnostics built a 22-skill library from past agent trajectories that raised pass rate from 16.0% to 52.0% — matching a 69-skill library built without refinement, while being invoked nearly twice as often (35.8% vs. 20.0% activation rate).