Self-Authored Skills Crashed One Model's Accuracy From 78% to Under 1%

Notes on EvoClawBench: Can Agents Learn Reusable Skills from Their Own Runs? (arXiv:2607.09711) — Zhiyuan Peng, Xin Yin, Chenhao Ying, Zhe Cui, Zixiang Ding, Zhenhua Liu, Jiang Wu, Yuan Luo · June 2026

Note published July 25, 2026 · written by SkillFed’s research pipeline from the paper above · how these notes are made

AI-assisted notes · reviewed by SkillFed Skill evolution Bridge: evolution × benchmarks

EvoClawBench asks a narrow, closed-loop question: can an agent turn evidence from its own task runs into a reusable skill that makes a fresh run better, after paying the cost of authoring it? The benchmark spans 100 tasks (502 sub-problems) across coding, data, office, security, operations, and domain-document workflows — finance, legal, healthcare, procurement and more. Every task runs through three conditions: a direct baseline with no skill involved, PreSkill (write a skill before ever attempting the task, then solve using only that skill), and PostSkill (attempt the task once, then summarize a skill from that run's evidence and solve again in a fresh workspace). Two agent runtimes, OpenClaw and nanobot, each ran five model backends — GPT-5.4, GPT-5.4 mini, Qwen3.6-Plus, DeepSeek-V4-Pro, and MiniMax-M2.7 — through all three conditions on identical tasks.

The choice of runtime scaffolding swamps the choice of model: OpenClaw stayed under 20% baseline accuracy across every backend, while nanobot ranged from 56.45% to 96.13% on the same tasks. Adding self-authored skills doesn't move scores in one direction. Nanobot's MiniMax-M2.7 rose from 90.97% to 94.50% under PostSkill, and its GPT-5.4 held above 96% throughout. But nanobot's DeepSeek-V4-Pro collapsed from 77.77% baseline to 4.80% under PreSkill and 0.99% under PostSkill, and OpenClaw's GPT-5.4 fell from 18.63% to 1.14% under PostSkill. The number of skills generated tracked none of this — batches of 20-plus skills produced both wins and wipeouts. Even the wins aren't free: skill workflows ran at token-efficiency ratios as low as 0.21-0.40 against baseline (roughly 2.5x to nearly 5x the token spend) for a slightly lower score. Wall-clock time only breaks even after 9 to 14 reuses of the same skill.

Key numbers

Benchmark scale	100 tasks / 502 sub-problems
OpenClaw baseline ceiling (all 5 models)	under 20%
nanobot DeepSeek-V4-Pro: Baseline→PreSkill→PostSkill	77.77% → 4.80% → 0.99%
nanobot MiniMax-M2.7 under PostSkill	90.97% → 94.50%
OpenClaw+Qwen3.6-Plus PostSkill token efficiency	0.30x baseline (~3.3x tokens)

What this means for skill authors

The sharpest result here isn't the headline collapse, it's the ablation buried in the appendix. On a 4-task subset, a real self-authored skill lifted mean score by 4.05 points — but a deliberately empty scaffold lifted it by 19.84 points, and an irrelevant skill by 1.52. The authors are direct about what that means: some of what looks like 'skill content working' is actually the reuse prompt, the context reset, and ordinary run-to-run variance. For anyone building a skill authoring pipeline, that's a warning against grading a skill on the run that produced it. A skill only earns trust from a fresh, independent execution — ideally several — not from the transcript it was distilled out of.

That's also the practical argument for treating skill provenance as a first-class signal rather than an afterthought. A skill an agent writes for itself, from one imperfect run, is unvalidated code shipped straight from an untested source — and this paper documents that code actively wrecking accuracy in specific runtime-model pairings, with no visible warning sign beforehand. Vetting skills before they're trusted — the way a catalog reviews an artifact once, independent of the agent that produced it — is a different bet than letting every agent regenerate and re-risk its own skill library on every task. SkillFed's dual-tool scanning approach to a shared catalog is built on exactly that distinction; this paper is evidence for why the distinction matters, not a citation of any SkillFed number.

There's a second, quieter lesson about cost. Even the skill runs that didn't hurt accuracy burned several times the tokens of the baseline. The authors calculate that wall-clock time alone needs 9 to 14 repeated uses of a skill before the authoring investment pays for itself — before counting token cost or dollars, which never broke even on their subset. A skill worth keeping has to clear a quality bar and an amortized-cost bar. Skills authored once and reused across many callers, rather than reauthored per agent per task, are the version of this that actually pays that cost down.

Skills related to this research

Socratic Teaching Scaffolds This skill teaches you to structure learning around strategic questioning and graduated support. It combines question ladders, misconception detectors, and worked-example fading to help learners build durable understanding while progressively removing scaffolding as competence grows. Use it to teach complex concepts, correct mental models, onboard team members, or design self-paced learning paths.★ 135 archify Archify transforms plain-language descriptions or JSON specifications into self-contained, explorable HTML diagrams for system architecture, workflows, API sequences, data pipelines, and state machines. Output includes interactive features like pan/zoom, theme switching, and optional trace animation, with exports to SVG, PNG, JPEG, WebP, and WebM.★ 7,642

Related notes

References

Peng et al., "EvoClawBench: Can Agents Learn Reusable Skills from Their Own Runs?", arXiv:2607.09711 (2026)
Jimenez et al., "SWE-bench: Can Language Models Resolve Real-World GitHub Issues?" (2024)
Zhou et al., "WebArena: A Realistic Web Environment for Building Autonomous Agents" (2024)
Mialon et al., "GAIA: A Benchmark for General AI Assistants" (2024)