The validation gate turns a 40% agent into an 89% one — skill-writing without it changes nothing

Notes on GRASP: Gated Regression-Aware Skill Proposer for Self-Improving LLM Agents (arXiv:2605.29668) — Johannes Moll, Jean-Philippe Corbeil, Jiazhen Pan, M. Hadamitzky, D. Rueckert, Lisa Adams, Keno K. Bressem · May 2026

Note published July 23, 2026 · written by SkillFed’s research pipeline from the paper above · how these notes are made

AI-assisted notes · reviewed by SkillFed Skill evolution

GRASP treats agent self-improvement as edits to a small, versioned skill library, not an ever-growing memory dump. When an agent fails, the failure gets a mechanism-specific label, and a proposer drafts several candidate fixes — add, modify, or remove a skill — aimed first at whichever failure type is most common. Before anything gets written into the library, each candidate has to survive a regression gate: it runs against a held-out probe of past-failing and past-passing episodes, and only sticks if it fixes more than it breaks and stays under a hard cap on new regressions. Testing spanned five base models — gpt-oss-120b, DeepSeek V4 Flash, Gemini 3.1 Flash Lite, GPT-4.1, GPT-5.4 — on two FHIR-based clinical agent benchmarks, then checked whether the mechanism holds outside medicine.

The gate is the entire effect. gpt-oss-120b jumps from 40.6% to 88.8% on MedAgentBench, 21 points past the best of five self-improvement baselines (Evo-MedAgent, at 67.8%), and every other model gains 17 to 40 points. Strip the gate out and let the same proposer write skills unchecked, and accuracy lands at 40.1% — statistically the no-skills baseline, since the writing was never the hard part. Drop just the hard regression ceiling while keeping a net-improvement check, and it falls to 81.8%, so the ceiling carries real weight on its own. The pattern holds on three of four non-clinical environments (ALFWorld +28.4, WebShop +20.6, DBBench +5.0) and goes flat (+0.9) only on OS Interaction, where actions don't recur in a checkable way. Frozen libraries transfer unevenly too: skills written by a stronger model lift a weaker executor past what it manages on its own, but skills from a weaker model drag a stronger executor below its own self-training baseline — an asymmetry no ungated baseline reproduces.

Key numbers

gpt-oss-120b on MedAgentBench	40.6% → 88.8%
Margin over best baseline	+21.0 pts vs Evo-MedAgent
Skill-writing with no gate	40.1% (≈ no-skills baseline)
Regression ceiling removed	81.8% (down from 88.8%)
Library size vs memory baselines	5.6k vs 34–53k tokens

What this means for skill authors

For anyone assembling a skill catalog, this is the uncomfortable part: how well a skill is written barely matters if nothing checks whether it breaks something else. A library that only ever appends — the pattern behind sequential and batch memory in this paper — degrades exactly the way vetting exists to catch. A dual-tool scan and a held-out check before a skill ships are different mechanisms pointed at the same failure GRASP names: an edit that fixes one case while silently breaking three others.

The cross-model transfer result reframes who gets to author a skill. A validated library written by a stronger model measurably helps a weaker one beyond its own self-improvement ceiling — but the reverse trade is a net loss next to just letting the weaker model train itself. That argues for treating skill authorship as a role distinct from skill execution: a catalog's value sits in who validated the skill and against which model, not just whether it happens to run.

The flat result on OS Interaction marks the real edge of this mechanism. Skill libraries earn their keep where a task repeats with a checkable structure — clinical order sets, embodied pick-and-place — because the gate needs a stable failure signature to check against. Open-ended action spaces never produce one, so there's nothing reliable to validate a candidate skill against. Expect this approach to compound in narrow, procedural domains and to do little in freeform ones.

Skills related to this research

playwright-verifier This skill automates accessibility compliance verification by testing whether your fixes actually resolve violations and don't create new ones. Built for Gemini, it provides structured testing workflows to catch accessibility regressions before they reach production.★ 370 statistical-analysis Conduct rigorous statistical tests—t-tests, ANOVA, chi-square, correlation, regression, and Bayesian methods—with systematic assumption verification and effect size reporting. The skill walks you through test selection, data inspection, assumption diagnostics, and APA-style write-ups so your analysis withstands peer review.★ 31,940 statsmodels Statsmodels is Python's statistical modeling library for fitting regression, generalized linear, time series, and discrete outcome models with full inference capabilities. It delivers detailed diagnostics, residuals analysis, hypothesis tests, and publication-ready output—ideal when coefficients and their uncertainty matter more than prediction alone.★ 31,940 statsmodels-stats statsmodels-stats provides regression modeling, hypothesis testing, and time series analysis for statistical workflows. Build OLS, logistic, and generalized linear models; run t-tests, ANOVA, and proportion tests; fit ARIMA and VAR models; and perform survival analysis with diagnostic plots and rigorous statistical reporting.★ 869

Related notes

References

Moll, J., Corbeil, J.-P., Pan, J., Hadamitzky, M., Rueckert, D., Adams, L., & Bressem, K.K. (2026). GRASP: Gated Regression-Aware Skill Proposer for Self-Improving LLM Agents. arXiv:2605.29668.
Shinn, N., Cassano, F., Gopinath, A., Narasimhan, K., & Yao, S. (2023). Reflexion: Language Agents with Verbal Reinforcement Learning. NeurIPS.
Zhao, A., Huang, D., Xu, Q., Lin, M., Liu, Y.-J., & Huang, G. (2024). ExpeL: LLM Agents Are Experiential Learners. AAAI.