Reward reuse, not just creation: +8.9 points scenario completion, -59% tokens

Notes on Reinforcement Learning for Self-Improving Agent with Skill Library (arXiv:2512.17102) — Jiong Wang, Qiaojing Yan, Yawei Wang, Yijun Tian, Soumya Smruti Mishra, Zhichao Xu, Megha Gandhi, Panpan Xu, Lin Lee Cheong · December 2025 · Annual Meeting of the Association for Computational Linguistics

Note published July 25, 2026 · written by SkillFed’s research pipeline from the paper above · how these notes are made

AI-assisted notes · reviewed by SkillFed Skill evolution

SAGE trains an LLM agent to build and reuse a library of executable skills by extending GRPO (Group Relative Policy Optimization) with two additions. Sequential Rollout chains two similar tasks from the same scenario into a single rollout, so skills the agent writes while solving the first task are still sitting in its skill library when it tackles the second — letting the reward signal from successful reuse flow back to the skill-writing step. A Skill-integrated Reward then stacks bonus credit on top of the normal pass/fail outcome reward — one bonus when a generated skill actually gets used by the next task, another when a task successfully reuses an earlier one. The agent writes in a DynaSaur-style unified format — a callable function it writes and immediately invokes, rather than calling raw APIs directly — so skill creation and task-solving happen in the same step, not as a separate post-hoc phase.

On AppWorld — 750 tasks across 250 three-task scenarios, scored with Scenario Goal Completion (SGC), which requires all three related tasks in a scenario to succeed — a Qwen2.5-32B agent trained with SAGE reaches 72.0% Task Goal Completion and 60.7% SGC on the Test-Normal split, averaging 12.1 steps and 1,475 tokens per task. A GRPO agent trained without any skill library manages only 69.2%/51.8%, at 16.4 steps and 3,613 tokens — SAGE's SGC edge (51.8% to 60.7%, 8.9 points) arrives alongside 26% fewer steps and 59% fewer tokens. Ablations trace the gain to the reward design, not just the presence of a skill library: swapping Skill-integrated Reward for a plain outcome-based or chain-based reward drops SGC to 55.4% and 56.6%; skipping supervised fine-tuning on expert trajectories before RL collapses SGC as low as 25.6%. A pure prompting-based skill library agent, with no SFT and no RL, manages only 19.6% SGC — proof that letting a model freely invent and reuse skills through prompting alone doesn't hold up.

Key numbers

SGC improvement over GRPO baseline (Test-Normal)	+8.9 points (51.8% → 60.7%)
Token reduction vs. GRPO baseline	-59% (3,613 → 1,475 tokens)
Interaction-step reduction vs. GRPO baseline	-26% (16.4 → 12.1 steps)
SGC with RL from base model, no SFT warm-start	25.6% (vs. 60.7% with SFT init)
AppWorld benchmark scale	750 tasks / 250 three-task scenarios

What this means for skill authors

The uncomfortable finding for anyone betting on prompt-only skill libraries: letting a model invent and reuse skills purely through instructions, with no training behind it, lands at 19.6% SGC. The framework doesn't pay off until reward gets explicitly tied to whether a generated skill is picked up and reused downstream, not just whether the immediate task passed — a curation problem as much as a generation problem, and exactly the gap between an agent hoarding self-written functions and a library worth trusting. There's a retrieval ablation buried in the same paper worth flagging: cheap Query N-gram matching hits 60.1% SGC, almost matching the oracle case where the correct scenario's skills are handed to the agent directly (60.7%), while semantic Query/Skill Embedding retrieval trails at 59.5% and 56.0%. Lexical overlap beating embeddings for skill retrieval is a useful, slightly contrarian data point for anyone building skill discovery.

The broader shape of the result tracks why a vetted, dual-tool-scanned catalog matters more as agents get better at self-improvement, not less: this paper's skills are agent-invented, unreviewed, and tested only inside AppWorld's simulator. SAGE's own ablations show reward design and evaluation-environment quality determine whether a skill library helps or actively hurts — the same open question a federation of externally-vetted skills exists to answer for skills an agent didn't write itself. Keep the scope honest: results come from one benchmark, task chains capped at two examples, and a base retrieval setup that assumes an oracle scenario label by default. Whether Sequential Rollout and Skill-integrated Reward generalize beyond AppWorld to other tool-using environments is still open.

Skills related to this research

fine-tuning-with-trl This skill teaches post-training techniques for aligning language models to human preferences. It covers supervised fine-tuning, direct preference optimization (DPO), and online reinforcement learning methods like RLOO and GRPO, with complete workflows and practical examples.★ 221,503 fine-tuning-with-trl Fine-tuning with TRL provides post-training methods to align language models with human preferences through multiple approaches. Train models on instruction data with SFT, optimize for preference alignment via DPO without a separate reward model, or run full RLHF pipelines combining supervised fine-tuning, reward modeling, and PPO optimization. GRPO offers memory-efficient online reinforcement learning for resource-constrained setups.★ 11,165 Unsloth Training Skill This skill teaches efficient LLM fine-tuning via Unsloth, covering GRPO (reinforcement learning with reward functions) and SFT (supervised training). Learn FP8 training for 60% VRAM reduction, automatic packing for 2–5x speedup, vision model tuning, mobile deployment via QAT, Docker containerization, and export to GGUF and other formats.★ 26 Unsloth Training Unsloth Training accelerates LLM fine-tuning through GRPO (reinforcement learning with reward functions) and SFT (supervised learning with input-output pairs). It cuts VRAM usage by 60% with FP8 training, speeds up mixed-length datasets via automatic packing, and supports vision model tuning, mobile deployment via ExecuTorch, and export to GGUF, Ollama, and vLLM.★ 8

Related notes

References

J. Wang, Q. Yan, Y. Wang, Y. Tian, S. S. Mishra, Z. Xu, M. Gandhi, P. Xu, L. L. Cheong, "Reinforcement Learning for Self-Improving Agent with Skill Library" (SAGE), arXiv:2512.17102, 2025.
Z. Shao et al., "DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models" (introduces GRPO), arXiv:2402.03300, 2024.
H. Trivedi et al., "AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents," ACL 2024.
D. Nguyen et al., "DynaSaur: Large Language Agents Beyond Predefined Actions," arXiv:2411.01747, 2024.
G. Wang et al., "Voyager: An Open-Ended Embodied Agent with Large Language Models," TMLR, 2024.