Reward reuse, not just creation: +8.9 points scenario completion, -59% tokens

Notes on Reinforcement Learning for Self-Improving Agent with Skill Library (arXiv:2512.17102) — Jiong Wang, Qiaojing Yan, Yawei Wang, Yijun Tian, Soumya Smruti Mishra, Zhichao Xu, Megha Gandhi, Panpan Xu, Lin Lee Cheong · December 2025 · Annual Meeting of the Association for Computational Linguistics

Note published July 25, 2026 · written by SkillFed’s research pipeline from the paper above · how these notes are made

SAGE trains an LLM agent to build and reuse a library of executable skills by extending GRPO (Group Relative Policy Optimization) with two additions. Sequential Rollout chains two similar tasks from the same scenario into a single rollout, so skills the agent writes while solving the first task are still sitting in its skill library when it tackles the second — letting the reward signal from successful reuse flow back to the skill-writing step. A Skill-integrated Reward then stacks bonus credit on top of the normal pass/fail outcome reward — one bonus when a generated skill actually gets used by the next task, another when a task successfully reuses an earlier one. The agent writes in a DynaSaur-style unified format — a callable function it writes and immediately invokes, rather than calling raw APIs directly — so skill creation and task-solving happen in the same step, not as a separate post-hoc phase.

On AppWorld — 750 tasks across 250 three-task scenarios, scored with Scenario Goal Completion (SGC), which requires all three related tasks in a scenario to succeed — a Qwen2.5-32B agent trained with SAGE reaches 72.0% Task Goal Completion and 60.7% SGC on the Test-Normal split, averaging 12.1 steps and 1,475 tokens per task. A GRPO agent trained without any skill library manages only 69.2%/51.8%, at 16.4 steps and 3,613 tokens — SAGE's SGC edge (51.8% to 60.7%, 8.9 points) arrives alongside 26% fewer steps and 59% fewer tokens. Ablations trace the gain to the reward design, not just the presence of a skill library: swapping Skill-integrated Reward for a plain outcome-based or chain-based reward drops SGC to 55.4% and 56.6%; skipping supervised fine-tuning on expert trajectories before RL collapses SGC as low as 25.6%. A pure prompting-based skill library agent, with no SFT and no RL, manages only 19.6% SGC — proof that letting a model freely invent and reuse skills through prompting alone doesn't hold up.