Diagnose, don't rewrite: three revision rounds take a skill from 36% to 62% success

Notes on SkillRevise: Improving LLM-Authored Agent Skills via Trace-Conditioned Skill Revision (arXiv:2606.01139) — Yuxuan Liu, Zhaochen Su, Lin Xie, Yuhao Zhang, Qing Zong, Jiahe Guo, Zhongwei Xie, Yiyan Ji, Yauwai Yim, Hongyu Luo, Xiyu Ren, Chenyu Ruan, and 2 more · May 2026

Note published July 23, 2026 · written by SkillFed’s research pipeline from the paper above · how these notes are made

AI-assisted notes · reviewed by SkillFed Skill evolution

Most skills start life one of two ways: an expert writes them by hand, or an LLM generates them in a single pass from a task description. Both are cold-start methods — neither one looks at what actually happens when an agent tries to execute the skill. SkillRevise closes that loop. It runs the current skill against a real task, captures the execution trace and verifier outcome, and turns that evidence into a structured diagnosis: what the verifier actually required, which specific behaviors caused the failure, and which parts of the skill already work and must be preserved. That diagnosis is matched against a fixed bank of seven general repair principles — reusable defect patterns like malformed schema fields or an unguarded tool failure — and only the principles whose evidence conditions are satisfied get bound to this repair. The revision itself is anchored: each edit specifies exactly what executor behavior it's supposed to change, not just what text to add.

The edited skill is then re-executed on the same task, and the framework keeps searching only until it finds the first version that actually passes the verifier — it doesn't default to whatever the last rewrite happened to be, and it only falls back to a heuristic utility score if nothing in the budget passes. Across three benchmarks (SkillsBench, SkillLearnBench-Random, SWE-Skills-Bench-Hard) and five different executor LLMs, three rounds of this loop consistently beat both no-skill execution and one-shot generation. On SkillsBench specifically, GPT-5.5's success rate moves from 36.05% with no skill, to 39.53% with one-shot generation, to 61.63% after three rounds of revision. A follow-up ablation isolates why: strip out the diagnosis step and success on the same benchmark collapses from 53/86 to 28/86 — the single largest drop of any component tested.

Key numbers

GPT-5.5 success on SkillsBench, no skill → 3 revision rounds	36.05% → 61.63%
Success rate w/o the diagnosis step (SkillsBench ablation)	61.6% → 32.6% (53/86 → 28/86)
One-shot generated skill vs. no skill at all (GPT-5.5, 206 tasks)	78/206 vs 79/206 — no better
ALFWorld tasks solved from a 10-principle bank absorbed over 25 calibration tasks	71/100
Benchmarks × executor LLMs × tasks evaluated	3 benchmarks × 5 LLMs × 206 tasks

What this means for skill authors

The finding worth sitting with is the one-shot number: a freshly LLM-generated skill (78/206) performed no better than giving GPT-5.5 no skill at all (79/206). For Opus-4.7 it gets stranger — a one-shot skill actually beat the framework's own unrevised initial draft, while that initial draft underperformed no-skill execution outright. A skill file that reads as syntactically competent — clear steps, sensible tool calls — is not evidence it helps. The only way the paper found to tell was to run it and check the verifier. That is an uncomfortable fact for anyone authoring or adopting skills without an execution step in the loop: readability is not reliability, and a plausible-looking skill can quietly cost you accuracy relative to just not having one.

This is the same reasoning behind SkillFed's dual-tool scanning approach to a vetted catalog: a skill's provenance and prose don't establish that it works, only checking it against real execution does. That's qualitatively the same posture SkillRevise takes toward its own diagnosis step. The paper's ablation makes the mechanism explicit: execution anchors, the preservation ledger, and the principle bank each contribute, but diagnosis — the specific act of mapping a verifier failure to a defect in the skill text — is what makes revision aim at the right target instead of just rewriting. For skill authors, that argues for building in a cheap verifier and a bounded number of repair attempts before publishing, rather than trusting a single generation pass.

The transfer result is the more speculative but interesting thread: skills revised against GPT-5.5's execution traces still improved outcomes when handed to Opus-4.7, Kimi-2.6, Qwen-3.6-Plus, and DeepSeek-V4-Pro, even though executor-specific revision remained stronger in every case. That's a partial answer to a real catalog question: does a skill tuned against one model's behavior degrade when a different model runs it? Here, no — not entirely. But the gap between transferred and native revision suggests some of what gets fixed is model-specific prompt convention rather than portable procedure. That matters for anyone deciding whether to re-diagnose a skill per executor or trust one revised version across a fleet.

Skills related to this research

humanize Humanize rewrites AI-generated text by disrupting the statistical patterns that detectors like Pangram and GPTZero rely on, then re-renders it in your authentic voice from a saved profile. Load your voice profile first (or generate one with the voice-profiler skill) to get YOU-specific output instead of generic humanization. For best results with classifiers, manually retype or verbally transcribe the output before submission.★ 2 humanize-ig humanize-ig transforms polished AI writing into casual Instagram voice by applying 10 core rules: lowercase everything, drop apostrophes, use fragments, abbreviate freely, and pack it into one dense paragraph. It scores your text against 16 AI detection patterns across statistical, deep learning, and document-level tiers, then rewrites accordingly. Load a voice profile for personalized output that carries your actual fingerprint.★ 2 bug-hunt-swarm Bug Hunt Swarm deploys four read-only sub-agents in parallel to investigate bugs, regressions, and crashes from different angles—reproduction scope, code paths, recent changes, and proof strategies. The main agent then synthesizes findings into ranked hypotheses with supporting evidence and the fastest path to confirm or disprove each theory, all without making code edits.★ 3,861 earnings-revision Monitor sell-side analyst consensus changes and earnings surprise patterns to generate investment signals across US and Hong Kong markets. The skill covers revision momentum, post-earnings announcement drift (PEAD), management guidance analysis, and earnings quality indicators—all documented alpha factors in equity markets.★ 28,096

Related notes

References

SkillRevise: Improving LLM-Authored Agent Skills via Trace-Conditioned Skill Revision, Liu et al., 2026 (arXiv:2606.01139)
SkillsBench: benchmarking how well agent skills work across diverse tasks, Li et al., 2026 (arXiv:2602.12670)
SWE-Skills-Bench: do agent skills actually help in real-world software engineering?, Han et al., 2026 (arXiv:2603.15401)
SkillLearnBench: benchmarking continual learning methods for agent skill generation on real-world tasks, Zhong et al., 2026 (arXiv:2604.20087)
Reflexion: language agents with verbal reinforcement learning, Shinn et al., NeurIPS 2023