Skill synthesis that checks its own work: +3 to +10 accuracy points, only 6% of skills still backfire

Notes on SkillGen: Verified Inference-Time Agent Skill Synthesis (arXiv:2605.10999) — Yuchen Ma, Yue Huang, Han Bao, Haomin Zhuang, Swadheen Shukla, Michel Galley, Xiangliang Zhang, S. Feuerriegel · May 2026

Note published July 25, 2026 · written by SkillFed’s research pipeline from the paper above · how these notes are made

SkillGen turns an agent's own rollouts into a single, human-readable skill file instead of asking a person to write one. It collects successful and failed trajectories from a base agent, then runs contrastive induction over both: failures are clustered, each is matched to its nearest successful neighbor, and the behavioral gap between the two is extracted as the reusable signal — not a paraphrase of what happened, but the specific delta that explains why one run succeeded and a similar one didn't. Candidate skills are written into a fixed three-part schema (task context, success pattern, failure-avoidance pattern) and pushed through a generation-verification-refinement loop before anything ships.

The verification step is the real contribution. SkillGen treats a candidate skill as an intervention: it replays the same instances with and without the skill and nets out repairs against regressions before accepting it, and a verification gate blocks anything that misses a minimum net-gain threshold. Across eight base models and nine benchmarks (ALFWorld, ScienceWorld, PubMedQA, Mind2Web, LiveCodeBench, MCPBench, ChemLLMBench, tau-Bench, SocialMaze), every model gained on average — open-weight models by 3.27 to 4.77 points, proprietary models by 4.79 to 10.08 points — beating prior skill-generation baselines (Trace2Skill, SkillX, EvoSkill, CoEvoSkills). Of 80 benchmark-model pairs, 50 improved, 25 held steady, and 5 still regressed even after the gate. Skills also showed real cross-model transfer: about 70% of cross-model comparisons were non-negative, and 42% gained more than 5 points on a model that never produced the skill in the first place.