One in Four Model-Generated Skills Backfires on the Agent Using It

Notes on From Raw Experience to Skill Consumption: A Systematic Study of Model-Generated Agent Skills (arXiv:2605.23899) — Zisu Huang, Jingwen Xu, Yifan Yang, Ziyang Gong, Qihao Yang, Muzhao Tian, Xiaohua Wang, Changze Lv, Xuemei Gao, Qi Dai, Bei Liu, Kai Qiu, and 4 more · May 2026

Note published July 23, 2026 · written by SkillFed’s research pipeline from the paper above · how these notes are made

AI-assisted notes · reviewed by SkillFed Skill evolution Bridge: evolution × benchmarks

A full-lifecycle test bed for model-generated agent skills — experience generation, skill extraction, skill consumption — runs across five domains (ALFWorld, SpreadsheetBench, SWE-bench-Verified, SEAL-0, BFCL-v4), with six models serving as consumers — GPT-5.4 and its mini variant, Gemini-3.1-Pro and Flash-Lite, Qwen3.5-35B and 9B — five of which (all but Qwen3.5-9B, excluded as an unreliable extractor) also act as extractors. Utility is measured directly as the performance delta a skill produces on held-out tasks versus no skill at all, rolled into an Extraction Efficacy score per extractor and a Target Evolvability score per consumer. Across every extractor-target-domain combination, skills help 75% of the time — but the other 25% show negative transfer, and the risk isn't evenly spread: ALFWorld skills backfire in 47% of pairings, while SpreadsheetBench and SWE-bench-Verified hold steady at 13%. Being a strong task performer doesn't make a model a good skill writer: on SpreadsheetBench, Gemini-3.1-Flash-Lite extracts the highest-value skills despite GPT-5.4 posting the stronger raw baseline, and the same pool of extractors pays off very differently depending on which model ends up consuming the skill.

The deeper problem: skill quality isn't legible from the text itself. Asked to pick the better of two skills just by reading them, an LLM judge scores 46.4% — indistinguishable from a coin flip — and on the pairs where the true performance gap is largest, accuracy drops to 15.8%, meaning the judge picks the worse skill more often than the better one. So what actually separates a high-utility skill from a dud? Three traits keep showing up: naming the specific failure mechanism and pairing it with an executable fix, giving step-level instructions tied to real domain objects rather than generic advice, and blacklisting the specific high-risk actions that cause damage — each trait alone predicts the better skill in a pair 64-66% of the time. Hand those three traits to the judge as a rubric and its accuracy jumps to 73.8%; fold them into the extractor's own system prompt as a compact meta-skill and it improves all nine tested extractor-domain combinations — +1.55 percentage points on average, up to +3.7 points on SpreadsheetBench. A generic, equally detailed 'write a good skill' rubric does the opposite: it makes things worse, at -0.59 points.

Key numbers

Negative transfer, all domains	25% of extractor-target pairs show Δ<0
Most fragile domain (ALFWorld)	47% of pairings backfire
Unguided LLM judge accuracy	46.4% (indistinguishable from random)
Judge accuracy on hardest pairs	15.8% (picks the worse skill more often)
Meta-skill gain after validated rubric	+1.55pp avg, up to +3.7pp on SpreadsheetBench

What this means for skill authors

A quarter of model-generated skills backfiring isn't a rounding error — it's a failure rate hiding behind a headline number (75% positive) that looks reassuring until you hit ALFWorld's 47%. Drop a skill into an agent's context on the assumption that 'extracted from real experience' means 'safe to use,' and you're gambling on which domain, which extractor, and which consumer you happen to have. The finding that skill utility runs independent of model scale or baseline task strength closes off the obvious shortcut: you can't just trust the bigger or better-performing model's output more.

The judge result is what should worry anyone who vets skills by reading them. 46.4% accuracy on well-written-versus-not is bad; 15.8% on the hardest pairs is actively anti-correlated with quality — a text reviewer would systematically promote the worse skill. That's the practical case for testing a skill against the agent that will actually run it, rather than scoring its prose: a catalog that vets by reading is vetting the wrong signal. It's the same logic behind treating a skill's runtime behavior, not its documentation, as what's worth scanning before a skill reaches a catalog.

For skill authors, the fix on offer is narrow and usable. Stop writing general best-practice advice. Name the exact way agents fail, and pair it with a concrete countermeasure. Tie steps to real objects and tools instead of abstractions. Blacklist the specific actions that cause damage. The paper shows this beats a longer, more thorough-sounding generic rubric — more text isn't the lever, specificity is. And because a skill's payoff depends on who consumes it as much as who wrote it, authors should expect to validate a skill against more than one downstream model before calling it finished.

Skills related to this research

Systematic Debugging Systematic Debugging teaches a disciplined 4-phase approach: root cause investigation, pattern analysis, hypothesis testing, and implementation. Rather than attempting quick fixes, this skill enforces tracing issues back to their source through careful error analysis, data flow tracking, and minimal testing before any code change.★ 17 Lesson Study Cycle Designer Lesson Study Cycle Designer structures the Japanese jugyō kenkyū process for teacher teams investigating student learning challenges. It generates a research theme, detailed cycle plan with meeting agendas, a research lesson designed for collaborative observation, and protocols focused on case students. The skill grounds itself in evidence from Stigler & Hiebert, Lewis, Perry & Murata, Dudley, and Takahashi & McDougal on how teachers develop shared professional knowledge through structured cycles of planning, teaching, observing, and analysing.★ 472 systematic-debugging This skill enforces a disciplined debugging process: establish a tight feedback loop that reproduces the exact symptom, gather evidence across system boundaries, trace data flow to its source, and form testable hypotheses before proposing any fix. By completing each phase sequentially—root cause investigation, pattern analysis, hypothesis testing, and verification—you eliminate guesswork and prevent quick patches from masking deeper issues.★ 221,503 systematic-debugging Systematic Debugging enforces a disciplined four-phase process: investigate root cause, analyze patterns, form and test hypotheses, then implement fixes. It stops you from guessing at solutions and catching yourself in cycles of failed patches that mask underlying problems.★ 6

Related notes

References

Huang, Xu, Yang, Gong, Yang, Tian, Wang, Lv, Gao, Dai, Liu, Qiu, Yang, Chen, Zheng & Luo, From Raw Experience to Skill Consumption: A Systematic Study of Model-Generated Agent Skills, arXiv:2605.23899 (2026)
Shridhar et al., ALFWorld: Aligning Text and Embodied Environments for Interactive Learning, arXiv:2010.03768 (2020)
Jimenez et al., SWE-bench: Can Language Models Resolve Real-World GitHub Issues?, arXiv:2310.06770 (2023)