Curated skills add 16 points of task success — up to 51.9 in healthcare

Notes on Agent Skill Evaluation and Evolution: Frameworks and Benchmarks (arXiv:2606.11435) — Kexin Ding, Yang Zhou, Can Jin, Fengfeng Tong, Mu Zhou, Dimitris N. Metaxas · June 2026

Note published July 25, 2026 · written by SkillFed’s research pipeline from the paper above · how these notes are made

This survey traces how agent skills — packaged, reusable procedures that let LLM agents execute domain tasks — move from one-off authoring to systematic evaluation and revision. Skill-improvement methods sort into four paradigms: execution feedback (single-run signals, where systems like SkillForge and CoEvoSkills compare executed behavior against a reference and separate failure diagnosis from rewriting), trajectory distillation (patterns pulled from many runs, as in SPARK and Trace2Skill), library-level compression that merges or prunes redundant skills, and reinforcement learning that scores skill value against task-level reward. The same survey catalogs six families of skill-centric benchmarks — utility, generation, retrieval and routing, safety auditing, software engineering, and real-world deployment — noting what each one measures and how large it is.

The benchmark census turns up a lopsided picture: utility and safety testing already spans 11 professional domains and 581 audited packages, while skill-generation benchmarks cover only 20 tasks across 15 sub-domains — the harder problem is the least measured. On SkillsBench, adding curated skills lifts average task pass rate by 16 percentage points, but the payoff swings wildly: +4.5 points in software engineering, +51.9 in healthcare. SkillRouter shows why skill content matters at match time — routing on name and description alone, instead of the full skill body, costs 31 to 44 percentage points of retrieval accuracy across an ~80,000-skill pool. No existing benchmark tracks a single skill across repeated evolution rounds, either, and success metrics stay binary — pass or fail — leaving token cost, latency, and error type largely unmeasured.