Agents Writing Their Own Skills Close Just 45% of the Gap to Human Authors

Notes on SkillLearnBench: Benchmarking Continual Learning Methods for Agent Skill Generation on Real-World Tasks (arXiv:2604.20087) — Shan Zhong, Yiming Lu, Jingjie Ning, Yi Wan, Lihang Feng, Y. Ao, Leonardo F. R. Ribeiro, Markus Dreyer, Sean Ammirati, Chenyan Xiong · April 2026

Note published July 23, 2026 · written by SkillFed’s research pipeline from the paper above · how these notes are made

SkillLearnBench is the first benchmark built to test whether an agent can write its own agent skills well enough to matter, rather than testing skills a human already wrote. Researchers from Carnegie Mellon and Amazon AGI assembled 20 verified tasks across 15 sub-domains — software engineering, information retrieval, productivity tools, data analytics, content creation, and utilities — pulled from a community skill-usage taxonomy. Every task is checked twice before it counts: an agent with no skill must fail it at least half the time over ten tries, and a human-authored reference skill must be able to solve it. Four continual learning methods for generating skills from a task description are then pitted against each other: one-shot generation, self-feedback (the agent critiques and rewrites its own skill after one failed attempt), teacher feedback (an outside evaluator gives guidance across up to two QA rounds), and Anthropic's structured Skill-Creator pipeline. Six different skill-writing LLMs do the generating — three Claude models, three Gemini models — while Claude Sonnet 4.6 stays fixed as the agent that actually executes whatever skill comes out.

Every method beats the no-skill baseline (10.17% task accuracy), but none comes close to human-authored skills (74.50%). The strongest method, self-feedback, tops out at 31.08% accuracy — closing only about 45% of that gap — while also using the fewest tokens (390K, versus teacher feedback's 528K). No method wins across every LLM or task category, and handing skill-writing to a stronger model doesn't reliably help: bigger models tend to write more precise but more prescriptive skills, hardcoding specific values that break the moment a task instance doesn't match them. The sharpest result concerns iteration itself. Run self-feedback for four rounds and coverage stays flat while trajectory alignment steadily drops — accuracy briefly rises after round one, then collapses. The term for this is recursive drift: without new information, the agent just reshuffles its own skill text. Teacher feedback, which injects a real external signal each round, keeps improving through round four instead.