A nano model beats a flagship one, 0.7758 vs 0.7620, by editing its own harness

Notes on SkillOpt-Lite: Better and Faster Agent Self-evolution via One Line of Vibe (arXiv:2607.03451) — Yifei Shen, Bo Li, Xinjie Zhang · July 2026

Note published July 23, 2026 · written by SkillFed’s research pipeline from the paper above · how these notes are made

Skill optimization here becomes a form of zeroth-order optimization: treat a skill document as a text parameter that execution feedback nudges, no gradient required. The authors map existing self-evolution methods — single-trace reflection, contrastive diagnosis, coordinate-wise edits, edit-budget decay — onto the classical zeroth-order toolbox, then note what separates an agent rollout from blind numerical perturbation: a rollout is a readable trajectory, planning steps and tool calls and error messages all visible, which turns optimization into targeted debugging rather than random search. Working from PAC-learning theory and an "everything is a file" systems philosophy, they distill the requirements for a skill-optimization loop that actually generalizes down to three: store each rollout as its own flat file instead of folding it into a pooled batch, mine consensus across several failures instead of patching from one (skip this and a stability coefficient in their analysis inflates the generalization error), and gate every candidate edit against a validation set that never overlaps the training failures. Strip out everything else — mini-batch reflection pooling, textual learning-rate decay, rejected-edit buffers — and what's left is SkillOpt-Lite: stage the trajectory, explore with shell tools, mine consensus and write a minimal patch, validate and gate. Four steps, run by a standard coding agent.

Across six benchmarks and five model scales, the stripped-down loop matches or beats the fully engineered SkillOpt baseline, and the gap is widest on tasks that demand multi-step reasoning. On LiveMath it beats full SkillOpt by 8.8 points on GPT-5.5 (73.6 vs. 64.8) and by 25.4 points on GPT-5.4-nano (55.7 vs. 30.3) — enough that the optimized nano model passes GPT-5.4 running the older pipeline. The Spreadsheet skill benchmark averages 69.7 versus SkillOpt's 57.1 across all five model scales, and on ALFWorld GPT-5.4-nano reaches 81.3% against 71.8%. On the more open-ended tasks, SearchQA and OfficeQA, the two pipelines land within about a point of each other, but SkillOpt-Lite gets there faster: most of its final validation score arrives within the first two or three optimization batches. Because the framework treats every agent artifact as an editable file, the authors push the same three-step loop onto the execution harness itself — tool definitions, retry logic, context windows — calling this HarnessOpt and gating it with compile checks, a five-example smoke test, and git-revertible toggles. On SpreadsheetBench, a separate 0-to-1 accuracy benchmark, HarnessOpt lets GPT-5.4-nano reach 0.7758 accuracy, edging out GPT-5.5 running a standard harness with full SkillOpt (0.7620): a smaller model overtaking a larger one purely by having a better-fitted environment.