Self-authored skills buy up to 2.9 points on out-of-distribution math for a 3% token tax

Notes on ARISE: Agent Reasoning with Intrinsic Skill Evolution in Hierarchical Reinforcement Learning (arXiv:2603.16060) — Yu Li, Rui Miao, Zhengling Qi, Tian Lan · March 2026

Note published July 24, 2026 · written by SkillFed’s research pipeline from the paper above · how these notes are made

ARISE trains a single policy to play two roles inside the same reinforcement-learning loop: a Skills Manager and a Worker. Before attempting a problem, the Manager scores every skill sitting in its cache by summing the policy's own log-probability of that skill's text conditioned on the query — no separate retriever model — and only prepends a skill to the Worker's prompt if the top score clears a confidence-gate threshold (0.35 by default). After the Worker's rollout, a dedicated extra rollout distills any successful trace into a compact five-field skill document — name, problem type, key insight, 2-3 step method, verification check — hard-capped at 220 characters. That document enters a two-tier cache-and-reservoir library through one of five operations: Add, Update, Evict, Load, Delete. A three-level reward — 2 for a correct answer that used a skill, 1 for a correct answer without one, 0 for any wrong answer regardless of skill use — biases GRPO's group-relative advantage toward skill-augmented rollouts whenever both appear in the same batch, so the same gradient step that improves reasoning also curates which skills survive.

Training runs on roughly 40K DeepScaleR problems (AMC, AIME, MATH, OlympiadBench); evaluation covers two base models — Qwen3-4B-Instruct and Phi-4-mini-instruct — on AMC 2023, AIME 2024/2025, and the four-domain Omni-MATH suite. ARISE beats plain GRPO, three stronger GRPO variants (Dr.GRPO, DAPO, GSPO), and two memory-augmented baselines (EvolveR, SimpleMem) on every benchmark for both models, measured as average Pass@1 over 32 runs. The advantage is largest off-distribution: Omni-MATH average accuracy rises 2.9 points on Qwen3-4B (23.9% to 26.8%) and 2.0 points on Phi-4-mini (9.4% to 11.4%), against gains of 1.2–2.6 points on the in-distribution competition sets. Ablations trace the effect to the reward itself: swapping the hierarchical signal for a plain binary reward erases most of the gain and collapses skill utilization from 73% to 31%, while removing the confidence gate barely moves accuracy but pushes indiscriminate skill injection up to 91%. The cost is modest — roughly 3% more tokens in total (66 extra prompt tokens per problem) and 16.6% more wall-clock time per training step, mostly spent generating the extra skill-summary rollout.