Diagnose, don't rewrite: three revision rounds take a skill from 36% to 62% success

Notes on SkillRevise: Improving LLM-Authored Agent Skills via Trace-Conditioned Skill Revision (arXiv:2606.01139) — Yuxuan Liu, Zhaochen Su, Lin Xie, Yuhao Zhang, Qing Zong, Jiahe Guo, Zhongwei Xie, Yiyan Ji, Yauwai Yim, Hongyu Luo, Xiyu Ren, Chenyu Ruan, and 2 more · May 2026

Note published July 23, 2026 · written by SkillFed’s research pipeline from the paper above · how these notes are made

AI-assisted notes · reviewed by SkillFed Skill evolution

Most skills start life one of two ways: an expert writes them by hand, or an LLM generates them in a single pass from a task description. Both are cold-start methods — neither one looks at what actually happens when an agent tries to execute the skill. SkillRevise closes that loop. It runs the current skill against a real task, captures the execution trace and verifier outcome, and turns that evidence into a structured diagnosis: what the verifier actually required, which specific behaviors caused the failure, and which parts of the skill already work and must be preserved. That diagnosis is matched against a fixed bank of seven general repair principles — reusable defect patterns like malformed schema fields or an unguarded tool failure — and only the principles whose evidence conditions are satisfied get bound to this repair. The revision itself is anchored: each edit specifies exactly what executor behavior it's supposed to change, not just what text to add.

The edited skill is then re-executed on the same task, and the framework keeps searching only until it finds the first version that actually passes the verifier — it doesn't default to whatever the last rewrite happened to be, and it only falls back to a heuristic utility score if nothing in the budget passes. Across three benchmarks (SkillsBench, SkillLearnBench-Random, SWE-Skills-Bench-Hard) and five different executor LLMs, three rounds of this loop consistently beat both no-skill execution and one-shot generation. On SkillsBench specifically, GPT-5.5's success rate moves from 36.05% with no skill, to 39.53% with one-shot generation, to 61.63% after three rounds of revision. A follow-up ablation isolates why: strip out the diagnosis step and success on the same benchmark collapses from 53/86 to 28/86 — the single largest drop of any component tested.

Key numbers

GPT-5.5 success on SkillsBench, no skill → 3 revision rounds	36.05% → 61.63%
Success rate w/o the diagnosis step (SkillsBench ablation)	61.6% → 32.6% (53/86 → 28/86)
One-shot generated skill vs. no skill at all (GPT-5.5, 206 tasks)	78/206 vs 79/206 — no better
ALFWorld tasks solved from a 10-principle bank absorbed over 25 calibration tasks	71/100
Benchmarks × executor LLMs × tasks evaluated	3 benchmarks × 5 LLMs × 206 tasks

What this means for skill authors

The finding worth sitting with is the one-shot number: a freshly LLM-generated skill (78/206) performed no better than giving GPT-5.5 no skill at all (79/206). For Opus-4.7 it gets stranger — a one-shot skill actually beat the framework's own unrevised initial draft, while that initial draft underperformed no-skill execution outright. A skill file that reads as syntactically competent — clear steps, sensible tool calls — is not evidence it helps. The only way the paper found to tell was to run it and check the verifier. That is an uncomfortable fact for anyone authoring or adopting skills without an execution step in the loop: readability is not reliability, and a plausible-looking skill can quietly cost you accuracy relative to just not having one.

This is the same reasoning behind SkillFed's dual-tool scanning approach to a vetted catalog: a skill's provenance and prose don't establish that it works, only checking it against real execution does. That's qualitatively the same posture SkillRevise takes toward its own diagnosis step. The paper's ablation makes the mechanism explicit: execution anchors, the preservation ledger, and the principle bank each contribute, but diagnosis — the specific act of mapping a verifier failure to a defect in the skill text — is what makes revision aim at the right target instead of just rewriting. For skill authors, that argues for building in a cheap verifier and a bounded number of repair attempts before publishing, rather than trusting a single generation pass.

The transfer result is the more speculative but interesting thread: skills revised against GPT-5.5's execution traces still improved outcomes when handed to Opus-4.7, Kimi-2.6, Qwen-3.6-Plus, and DeepSeek-V4-Pro, even though executor-specific revision remained stronger in every case. That's a partial answer to a real catalog question: does a skill tuned against one model's behavior degrade when a different model runs it? Here, no — not entirely. But the gap between transferred and native revision suggests some of what gets fixed is model-specific prompt convention rather than portable procedure. That matters for anyone deciding whether to re-diagnose a skill per executor or trust one revised version across a fleet.

Skills related to this research

ultraqa UltraQA automates quality assurance by cycling through test execution, failure diagnosis, and targeted fixes until your specified goal succeeds. It supports tests, builds, linting, type checking, or custom verification patterns, with a maximum of five cycles and early exit on repeated failures. The workflow coordinates diagnosis and remediation across multiple agents to streamline the path to passing quality gates.★ 38,122 lwy-prompt-optimizer This skill acts as a prompt coach, evaluating every instruction you send to an AI system across seven key dimensions: clarity, specificity, context, structure, examples, constraints, and completeness. It scores each area, identifies gaps, and delivers a rewritten version designed to produce better results on the first try. Works with any prompt type—code generation, writing, analysis, or creative tasks.★ 2 seo-traffic-diagnosis This skill helps you troubleshoot organic traffic declines by systematically analyzing technical health, search ranking shifts, and content performance. It guides you through root-cause investigation to pinpoint whether issues stem from indexing problems, algorithm updates, competitor activity, or content quality gaps.★ 500 doc-validator doc-validator enforces consistency rules across your entire SDD documentation corpus—catching broken links, orphaned artifacts, cumulative-tag gaps, duplicate IDs, and traceability gaps across all 8 layers. It runs declarative checks against your corpus structure and can optionally repair safely-fixable issues like missing reciprocal links or stale metadata. Use it before releases or after batch generation to ensure your documentation chain stays machine-verifiable and traceable.★ 16

Related notes

References

SkillRevise: Improving LLM-Authored Agent Skills via Trace-Conditioned Skill Revision, Liu et al., 2026 (arXiv:2606.01139)
SkillsBench: benchmarking how well agent skills work across diverse tasks, Li et al., 2026 (arXiv:2602.12670)
SWE-Skills-Bench: do agent skills actually help in real-world software engineering?, Han et al., 2026 (arXiv:2603.15401)
SkillLearnBench: benchmarking continual learning methods for agent skill generation on real-world tasks, Zhong et al., 2026 (arXiv:2604.20087)
Reflexion: language agents with verbal reinforcement learning, Shinn et al., NeurIPS 2023