One shared reward signal pushes skill-augmented agents to 97.5% success, beating every tested baseline — skill-augmented or skill-free — by up to 20 points

Notes on Skill1: Unified Evolution of Skill-Augmented Agents via Reinforcement Learning (arXiv:2605.06130) — Yaorui Shi, Yuxin Chen, Zhengxi Lu, Yuchun Miao, Shugui Liu, Qi Gu, Xunliang Cai, Xiangjun Wang, An Zhang · May 2026

Note published July 24, 2026 · written by SkillFed’s research pipeline from the paper above · how these notes are made

AI-assisted notes · reviewed by SkillFed Skill evolution

Skill1 trains one policy, via reinforcement learning, to run all three stages of a skill library lifecycle end to end: it writes a text query and re-ranks candidates to pick a skill, solves the task conditioned on that pick, then distills a new skill out of its own trajectory. The departure from prior work sits in credit assignment. Prior systems wire each stage to its own reward source — a heuristic match score for selection, a self-assessed quality score for distillation, task outcome for utilization — training three objectives that never talk to each other. Skill1 derives every signal from one binary task-outcome value instead, split into a low-frequency trend (an exponential moving average of each skill's success rate that supervises re-ranking through an NDCG-style ranking reward) and a high-frequency variation (current outcome minus the best retrieved skill's trend, which only pays out when a distilled skill beats the library's existing ceiling). One GRPO update per rollout batch trains query generation, re-ranking, and distillation together.

On ALFWorld, Skill1 reaches 97.5% average success, ahead of the strongest prior skill-augmented RL baseline (RetroAgent, 94.9%) by 2.6 points and the strongest skill-free RL baseline (GiGPO, 90.8%) by 6.7 points; on WebShop it scores 89.7 (82.9% success), again best among tested methods. The ablations are the more interesting result: pulling the skill library out entirely costs 16.6 points (down to 80.9%), dropping only the selection signal costs 5.7, dropping only distillation costs 5.1, and zeroing both auxiliary objective weights at once costs 7.3 — worse than either removal alone, the paper's evidence that the three capabilities are coupled, not merely additive. Training curves show selection precision converging first (0.95 by step 20), with utilization and distillation catching up afterward (0.8 by step 60), and the growing library adds real overhead: 1.3–1.7x the per-step wall-clock time of skill-free GRPO.

Key numbers

ALFWorld avg. success (Skill1)	97.5%
Margin over skill-free RL baseline (GRPO, 77.6%)	+19.9 pts
WebShop score / success rate	89.7 / 82.9%
Ablation: remove skill library entirely	80.9% (–16.6 pts)
Training overhead vs. skill-free GRPO	1.3–1.7x slower per step

What this means for skill authors

For anyone maintaining a library of reusable agent skills, the transferable finding isn't the RL machinery, it's the failure mode it exposes. A library that keeps accumulating entries without an active selection signal degenerates in a specific way: usage concentrates on a handful of popular skills while the rest sit unused, even as the total count keeps climbing and looks healthy from the outside. That's a distribution problem, not a training-algorithm problem, and it applies just as much to a hand-curated catalog as to a self-distilling one. A library nobody actively re-ranks is a library that quietly narrows itself in practice.

The more portable idea is how Skill1 gates new submissions: a distilled skill only earns reward when it beats the best matching skill already in the library, not simply when the task succeeds. That single design choice is why the resulting library covers close to twice as many distinct, frequently-used skills spread across a wider strategy space, instead of accumulating near-duplicates of whatever already scores well. It's the computational version of a problem every vetted skill catalog has to solve one way or another — SkillFed's dual-tool scanning and human review exist for the same reason: keep a growing corpus from filling up with redundant or low-quality near-misses instead of genuinely new coverage.

Worth keeping the scope honest: two text-based benchmarks, one 7B base model, a library capped at 5,000 entries — a cap the authors flag directly as unresolved. The coupling result holds convincingly inside that box; whether it survives at higher skill-catalog diversity, or when the library mixes self-distilled skills with human-authored ones, is untested by this paper.

Skills related to this research

fine-tuning-with-trl This skill teaches post-training techniques for aligning language models to human preferences. It covers supervised fine-tuning, direct preference optimization (DPO), and online reinforcement learning methods like RLOO and GRPO, with complete workflows and practical examples.★ 221,503 fine-tuning-with-trl Fine-tuning with TRL provides post-training methods to align language models with human preferences through multiple approaches. Train models on instruction data with SFT, optimize for preference alignment via DPO without a separate reward model, or run full RLHF pipelines combining supervised fine-tuning, reward modeling, and PPO optimization. GRPO offers memory-efficient online reinforcement learning for resource-constrained setups.★ 11,165 Unsloth Training Skill This skill teaches efficient LLM fine-tuning via Unsloth, covering GRPO (reinforcement learning with reward functions) and SFT (supervised training). Learn FP8 training for 60% VRAM reduction, automatic packing for 2–5x speedup, vision model tuning, mobile deployment via QAT, Docker containerization, and export to GGUF and other formats.★ 26 Unsloth Training Unsloth Training accelerates LLM fine-tuning through GRPO (reinforcement learning with reward functions) and SFT (supervised learning with input-output pairs). It cuts VRAM usage by 60% with FP8 training, speeds up mixed-length datasets via automatic packing, and supports vision model tuning, mobile deployment via ExecuTorch, and export to GGUF, Ollama, and vLLM.★ 8

Related notes

References

Yaorui Shi et al., "Skill1: Unified Evolution of Skill-Augmented Agents via Reinforcement Learning," arXiv:2605.06130 (2026).
Z. Shao et al., "DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models," arXiv:2402.03300 (2024) — introduces GRPO, the RL algorithm Skill1's joint update builds on.
M. Shridhar et al., "ALFWorld: Aligning Text and Embodied Environments for Interactive Learning," ICLR 2021 — one of the two environments used to evaluate Skill1.
S. Yao et al., "WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents," 2022 — the second evaluation environment.