Failure-only skill revision helps 1 model in 10, hurts the other 8

Notes on SkillEvolBench: Benchmarking the Evolution from Episodic Experience to Procedural Skills (arXiv:2605.24117) — Yi Lei, Zhongwei Wan, Jiankun Zhang, Samiul Alam, Zixuan Zhong, Peizhou Huang, X. Wang, Jingxuan Zhang, Donghao Zhou, Yunta Hsieh, Zhihao Dou, Hui Shen, and 4 more · May 2026

Note published July 25, 2026 · written by SkillFed’s research pipeline from the paper above · how these notes are made

SkillEvolBench tests something more specific than whether skills help an agent — whether an agent can turn its own one-off experience into a skill worth keeping. The benchmark spans 180 tasks across six real-world environments: code debugging, tool and API orchestration, data processing, document parsing, research synthesis, and communication/scheduling. Tasks are grouped into families that separate acquisition roles — canonical, enriched, variant — from three frozen deployment roles built to stress a skill after the fact: context shift, adversarial shortcuts, and skill composition. Agents write to and revise an external skill library during acquisition under outcome- and process-based verifier feedback; the library is then frozen before deployment tasks run, so nothing learned during evaluation can leak back into the skill being tested. Ten model configurations across three agent harnesses — Claude Code, Codex CLI, Gemini CLI — run self-generated and curated skill-writing policies against no-skill and raw-episodic-trajectory replay controls.

The headline diagnostic is a direct comparison: distilled skills versus just replaying the raw trajectories they were distilled from. It comes out lopsided toward the raw record — agents given untouched episodic replay beat agents given a written-up skill on most model and metric combinations, a sign that today's abstraction step discards contextual detail the task still needed. Update policy compounds the problem, and even the best case is shaky: rewriting the skill after every acquisition attempt is the healthiest policy tested, and it still only improves deployment success for 5 of 10 model configurations (2 tie, 3 lose); revising only after failures — arguably the more common real-world habit — helps just 1 configuration and hurts 8. Composition is the sharpest failure mode: in the communication-and-scheduling environment, success on composed-skill tasks sits at 0.0% under every condition tested, curated or self-generated alike. Bigger skill libraries don't help either — forcing larger optional resource bundles grows the library without reliably raising deployment success, because the added material tends to encode the specific episode rather than a procedure that generalizes.