Testing a Skill Before It's Banked Lifts WebShop Success From 72.7% to 78.1%

Notes on Co-Evolving Skill Generation and Policy Optimization (arXiv:2606.08755) — Zhiwei Zhang, Yudi Lin, N. Kuang, Lin Wu, Xiaomin Li, Songtao Liu, Fenglong Ma · June 2026

Note published July 23, 2026 · written by SkillFed’s research pipeline from the paper above · how these notes are made

AI-assisted notes · reviewed by SkillFed Skill evolution Bridge: evolution × retrieval

The working assumption behind skill-augmented agents — write a reusable procedural note into a retrievable skill bank, retrieve it on a later task — is that a stronger model writes a more trustworthy skill. Zhang and coauthors test that assumption directly, and it fails: skills GPT-5.4 writes while training agents on ALFWorld and WebShop carry a mean marginal utility near zero, even though a real subset of them helps and the rest actively hurts. The catch is that once a skill lands in the bank, its effect on later rollouts tangles up with every other skill retrieved alongside it — there's no clean signal for what any single skill contributed. Their framework, SAPO (Skill-Augmented Policy Optimization), tests each candidate before it reaches long-term memory. For a given task and its currently retrieved skills, SAPO splits the normal rollout budget into two matched halves: base rollouts run under the existing skill context alone, skill-augmented rollouts run under that same context plus one new candidate drawn from the base trajectories. The reward gap between the two halves is the candidate's marginal value in that exact context, computed from rollouts the training loop was already generating — validation costs nothing extra.

A candidate has to clear all three bars to get promoted to long-term memory: positive utility, ranked in the batch's top 20%, and dissimilar enough from what's already banked. Fall short on any of them and it's dropped before it can pollute retrieval. The same utility number then trains the policy itself to generate skills — an asymmetric objective reinforces positive-utility skills and suppresses harmful ones harder, cutting the need for repeated calls to a proprietary model. The policy's own likelihood of producing a given skill becomes a reusable score for pruning stale entries and for retrieval-time reranking as training goes on. Across ALFWorld and WebShop, this lifts WebShop success from 72.7% (SkillRL) to 78.1% and ALFWorld's full-task average from 89.9% to 92.2%. Across seven search-augmented QA benchmarks the average moves from 45.5% to 47.8%. Ablations show every piece earns its place: pull out just the utility-weighted generator training and WebShop success drops back to 73.4%.

Key numbers

Mean marginal utility of GPT-5.4-generated skills during training	≈0 (near zero)
WebShop success rate, SAPO vs. SkillRL	78.1% vs. 72.7%
ALFWorld all-task success, SAPO vs. SkillRL	92.2% vs. 89.9%
Search-augmented QA average across 7 benchmarks, SAPO vs. SkillRL	47.8% vs. 45.5%
WebShop success without utility-weighted generator training (ablation)	73.4%, down from 78.1%

What this means for skill authors

The number worth sitting with isn't the leaderboard result, it's the near-zero average marginal utility. A frontier model asked to distill a skill from a failed trajectory produces something useful about as often as something useless or actively misleading, and nothing about the model's cost or reputation predicts which in advance. That's a direct warning for any workflow that lets an agent mint its own skills at runtime: a well-written, plausible-sounding skill is not evidence of a good skill. The only test that means anything is whether it moves the reward for the task at hand, measured against what's already sitting in the retrieval context. SkillFed's dual-tool scanning at ingestion runs on the same premise in a different setting: a skill has to demonstrate something before it enters the catalog, not just describe itself well.

The more transferable idea is that utility is contextual, not intrinsic. A skill isn't good or bad in the abstract; it's good or bad relative to whatever else is already retrieved for that query, and its value can decay once the base policy stops needing the crutch. Static skill catalogs face a milder version of the same problem — a skill vetted once, in isolation, can still turn out redundant or stale next to whatever else a user's agent already has loaded. This paper shows that assumption breaking down inside online RL: vetting has to be an ongoing, context-aware measurement, not a one-time gate.

There's a design lesson in how cheaply the check gets bought, too. The validation trick works because it reuses rollouts the training loop needed anyway — it's not 'run twice as many trajectories,' it's 'split the ones you already budgeted into a matched pair.' That's the standard worth holding any skill-quality signal to, RL-native or catalog-side: if checking quality costs nearly as much as the thing it protects, most systems will quietly skip the check.

Skills related to this research

Photo Restoration Photo Restoration harnesses AI-powered models to repair physical damage like scratches and tears, recover faded colors, remove water stains, and colorize black and white photographs. The skill handles multiple restoration tasks—from grain removal to facial enhancement—bringing degraded family photos and archival images back to life in a single request.★ 24 Style Transfer Style Transfer reimagines your photos as artworks by applying techniques from famous painters, traditional media, and digital illustration styles. Choose from impressionist, cubist, watercolor, oil, anime, and custom reference styles while preserving your image's core composition and subject matter.★ 24

Related notes

References

Zhang, Z., Lin, Y., Kuang, N., Wu, L., Li, X., Liu, S., & Ma, F. (2026). Co-Evolving Skill Generation and Policy Optimization. arXiv:2606.08755.
Xia, P. et al. (2026). SkillRL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learning. arXiv:2602.08234.
Lu, Z. et al. (2026). Skill0: In-Context Agentic Reinforcement Learning for Skill Internalization. arXiv:2604.02268.
Tu, S. et al. (2026). Dynamic Dual-Granularity Skill Bank for Agentic RL (D2Skill). arXiv:2603.28716.
Shao, Z. et al. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (GRPO). arXiv:2402.03300.