One in Four Model-Generated Skills Backfires on the Agent Using It

Notes on From Raw Experience to Skill Consumption: A Systematic Study of Model-Generated Agent Skills (arXiv:2605.23899) — Zisu Huang, Jingwen Xu, Yifan Yang, Ziyang Gong, Qihao Yang, Muzhao Tian, Xiaohua Wang, Changze Lv, Xuemei Gao, Qi Dai, Bei Liu, Kai Qiu, and 4 more · May 2026

Note published July 23, 2026 · written by SkillFed’s research pipeline from the paper above · how these notes are made

A full-lifecycle test bed for model-generated agent skills — experience generation, skill extraction, skill consumption — runs across five domains (ALFWorld, SpreadsheetBench, SWE-bench-Verified, SEAL-0, BFCL-v4), with six models serving as consumers — GPT-5.4 and its mini variant, Gemini-3.1-Pro and Flash-Lite, Qwen3.5-35B and 9B — five of which (all but Qwen3.5-9B, excluded as an unreliable extractor) also act as extractors. Utility is measured directly as the performance delta a skill produces on held-out tasks versus no skill at all, rolled into an Extraction Efficacy score per extractor and a Target Evolvability score per consumer. Across every extractor-target-domain combination, skills help 75% of the time — but the other 25% show negative transfer, and the risk isn't evenly spread: ALFWorld skills backfire in 47% of pairings, while SpreadsheetBench and SWE-bench-Verified hold steady at 13%. Being a strong task performer doesn't make a model a good skill writer: on SpreadsheetBench, Gemini-3.1-Flash-Lite extracts the highest-value skills despite GPT-5.4 posting the stronger raw baseline, and the same pool of extractors pays off very differently depending on which model ends up consuming the skill.

The deeper problem: skill quality isn't legible from the text itself. Asked to pick the better of two skills just by reading them, an LLM judge scores 46.4% — indistinguishable from a coin flip — and on the pairs where the true performance gap is largest, accuracy drops to 15.8%, meaning the judge picks the worse skill more often than the better one. So what actually separates a high-utility skill from a dud? Three traits keep showing up: naming the specific failure mechanism and pairing it with an executable fix, giving step-level instructions tied to real domain objects rather than generic advice, and blacklisting the specific high-risk actions that cause damage — each trait alone predicts the better skill in a pair 64-66% of the time. Hand those three traits to the judge as a rubric and its accuracy jumps to 73.8%; fold them into the extractor's own system prompt as a compact meta-skill and it improves all nine tested extractor-domain combinations — +1.55 percentage points on average, up to +3.7 points on SpreadsheetBench. A generic, equally detailed 'write a good skill' rubric does the opposite: it makes things worse, at -0.59 points.