Route skill treatment by task difficulty, not skill category: +13.2 points OOD success

Notes on Skill0.5: Joint Skill Internalization and Utilization for Out-of-Distribution Generalization in Agentic Reinforcement Learning (arXiv:2605.28424) — Jiapeng Zhu, Jianxiang Yu, Yibo Zhao, Chengcheng Han, Qingdong Gu, Xunliang Cai, Xiang Li, Weining Qian · May 2026

Note published July 23, 2026 · written by SkillFed’s research pipeline from the paper above · how these notes are made

Skill0.5 stops treating every agent skill the same way. The usual binary — full skill externalization, where skills sit in the prompt at ever-growing token cost, versus full internalization, where they're baked into weights at ever-growing risk of overfitting — gets replaced with a router that reassigns each task in real time, based on the model's measured pass rate over several sampled rollouts. Tasks with zero successes go through privileged distillation: the model is rolled out first with an enriched prompt containing general, domain-agnostic skills, its successful trajectories become a teacher target, and a token-level Jensen-Shannon divergence loss trains the ordinary skill-free policy to match that teacher's output distribution — folding general strategy into the weights without paying for it at inference. Tasks with middling success get standard GRPO to push up raw success rate. Tasks that are already succeeding often — precisely where an agent is most likely to have learned a shortcut instead of actually reading the retrieved skill — get re-run with the task-specific skill deleted from the prompt; if removing it barely hurts performance, that gap gets folded into the advantage as a penalty that suppresses the shortcut. Everything runs on Qwen2.5-7B-Instruct with GRPO (group size 8) and skill retrieval via Qwen3-Embedding-0.6B.

On ALFWorld (six household task types, three held out as out-of-distribution domains) and WebShop (product-search goals split across seven categories, three held out as OOD), Skill0.5 beats the strongest skill-based RL baseline, SkillRL, by 2.3 points in-distribution and 13.2 points OOD on ALFWorld (93.1% vs 90.8% ID, 58.5% vs 45.3% OOD), and by 2.1 and 3.9 points respectively on WebShop. The in-distribution gain is incremental; the OOD gain is where the split treatment earns its keep. The two halves of the mechanism fail differently when removed. Drop the anti-shortcut probe and keep only internalization, and ID performance stays close to the full model but OOD generalization caps out at a noticeably lower ceiling. Drop internalization and keep only the utilization penalty, and both splits fall — a model with no distilled general-reasoning foundation can't act on a utilization signal it hasn't learned to use yet. The paper also traces a concrete shortcut learning failure in SkillRL's own training curve: its in-distribution success rate keeps climbing while its OOD accuracy declines in later training steps.