Deployment Failure, Not Self-Reflection, Drives Two-Thirds of a 13-Point Skill Gain

Notes on SkillEvolver: Skill Learning as a Meta-Skill (arXiv:2605.10500) — Ge Zhang, Erle Zhu, Jinfeng Zhou, Caiyan Jia, Hongning Wang · May 2026

Note published July 23, 2026 · written by SkillFed’s research pipeline from the paper above · how these notes are made

SkillEvolver treats skill-authoring as a closed loop, not a one-time artifact — and it never touches model weights; the update target is the skill's own prose and code. A meta-skill, loaded through the same CLI interface as any domain skill, drives an authoring agent through repeated cycles of exploring, patching, and auditing a candidate skill. Unlike trace-distillation methods that mine hundreds of pre-collected trajectories per domain, it works from a handful of trials on one newly arrived task. Each cycle spawns several trials built around genuinely different high-level strategies rather than resampled temperature, runs a contrastive skill update that isolates missing guidance by comparing traces that succeeded against ones that failed, and patches the artifact directly. The refinement signal comes specifically from watching a separate, fresh Domain-Skill Agent try to use the already-shipped skill and fail — not from the authoring agent grading its own exploration. Before a revision ships, an independent auditor session with no access to validation data runs nine mechanical checks for overfitting and deployment failure, including a silent-bypass check for skills that read as valid content but are never actually invoked at runtime.

Across 83 SkillsBench tasks spanning 15+ domains, two rounds of this loop lift accuracy from a 29.9% no-skill floor and a 43.6% human-curated baseline to roughly 57%. A single authoring pass alone reaches 48.2%; the second, deployment-grounded refinement pass supplies most of the remaining climb — one extra iteration accounts for about two-thirds of the total gain over curated skills. The improvement is lopsided in a useful way: it concentrates exactly where curated skills fail hardest or actively hurt, while human-authored skills still win outright on roughly a quarter of the 83 tasks, typically ones built around a domain-specific convention a person encodes into prose more precisely than a handful of trials can. On three KernelBench GPU-kernel tasks, the same loop lifts mean speedup from 1.16x to 1.51x, with the recurrent-network task alone climbing from 1.33x to 2.23x.