Train the skill, not the model: frozen GPT-4o-mini jumps from 6% to 42% on GAIA
Notes on Skill-R1: Agent Skill Evolution via Reinforcement Learning (arXiv:2605.09359) — Yash Vishe, Rohan Surana, Xunyi Jiang, Zihan Huang, Xintong Li, N. Kuang, Tong Yu, Ryan A. Rossi, Jingbo Shang, Julian McAuley, Junda Wu · May 2026
Note published · written by SkillFed’s research pipeline from the paper above · how these notes are made
AI-assisted notes · reviewed by SkillFed Skill evolutionSkill-R1 leaves the task-solving model frozen and trains something else entirely: a lightweight editor that rewrites the agent skill itself, generation after generation. Each round, the current skill induces a batch of rollouts from the frozen model (GPT-4o-mini here), a verifier scores them, and the editor reads those rollouts plus their pass/fail outcomes to write the next revision. Training uses a bi-level GRPO objective — an intra-generation term does the standard group-relative comparison among rollouts that share a skill, while an inter-generation term only rewards a revision if the population's mean reward actually climbed from the last round. Gradients never touch the task model, which is why the method works identically on open-weight or closed-source models — the paper calls this black-box compatibility.
On GAIA's 165 tasks, frozen GPT-4o-mini with no skill at all solves 6.1% of them. Give it a skill and train the editor with Skill-R1, and that jumps to 41.8% — 12.1 points above standard GRPO applied to the same editor (29.7%). The gain concentrates where it matters most: Level-3, the hardest tier, goes from 0% solved to 38.5%. WebWalker moves the same direction at smaller scale (26.0% vs. 22.0% for vanilla GRPO), but the comparison also exposes a limit — running the multi-generation rollout loop with no gradient updates at all isn't free. That untrained, inference-only editor actually underperforms vanilla GRPO on WebWalker (19.0% vs. 22.0%), even though it edges ahead on GAIA. And most of the gain lands early: by generation three of five, accuracy is already close to its final value; generations four and five mostly tighten consistency, with mean reward climbing from 0.06 to 0.44 and accuracy from 9.3% to 50.0% end to end.
Key numbers
| GAIA accuracy, no-skill vs. trained Skill-R1 | 6.1% → 41.8% |
| Gain over vanilla GRPO on GAIA | +12.1 points |
| GAIA Level-3 (hardest tier) accuracy | 0.0% → 38.5% |
| Gain over vanilla GRPO on WebWalker | +4.0 points (22.0% → 26.0%) |
| Untrained (inference-only) editor vs. vanilla GRPO on WebWalker | 19.0% vs 22.0% |
Skills related to this research
Related notes
- One shared reward signal pushes skill-augmented agents to 97.5% success, beating every tested baseline — skill-augmented or skill-free — by up to 20 points →
- Testing a Skill Before It's Banked Lifts WebShop Success From 72.7% to 78.1% →
- Reward reuse, not just creation: +8.9 points scenario completion, -59% tokens →
- Skill pass rates jump from 33.6% to 88.0% — with zero gradient steps →
- Self-authored skills buy up to 2.9 points on out-of-distribution math for a 3% token tax →
- 92.5% Skill Reuse From 816 Tokens →
- Deployment Failure, Not Self-Reflection, Drives Two-Thirds of a 13-Point Skill Gain →
- Checking what a value is for, not just whether it changed, cuts drift false alarms from 40% to zero →
- Skill synthesis that checks its own work: +3 to +10 accuracy points, only 6% of skills still backfire →
References
- Skill-R1: Agent Skill Evolution via Reinforcement Learning (arXiv:2605.09359)
- Shao et al., DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (arXiv:2402.03300) — source of the GRPO objective
- Mialon et al., GAIA: A Benchmark for General AI Assistants (ICLR 2024) — primary evaluation benchmark
- Wu et al., WebWalker: Benchmarking LLMs in Web Traversal (ACL 2025) — second evaluation benchmark
- Schulman et al., Proximal Policy Optimization Algorithms (arXiv:1707.06347) — RL baseline lineage