The right skill text for one model is the wrong one for another — model-aware rewriting recovers up to 25.8 points

Notes on Skill is Not One-Size-Fits-All: Model-Aware Skill Alignment for LLM Agents (arXiv:2605.30723) — Jianxiang Yu, Jiapeng Zhu, Bochen Lin, Qi Cui, Zichen Ding, Xiang Li · May 2026

Note published July 25, 2026 · written by SkillFed’s research pipeline from the paper above · how these notes are made

Across four Qwen3 backbones (4B, 8B, 14B, and 32B parameters), the same skill text does not transfer cleanly from one model to another. A controlled granularity study on ALFWorld makes the point sharply: moderate-detail skills help the 4B model, more detail helps the 32B model, and all three skill variants tested actually reduce the 8B model's success rate compared to giving it no skill at all. MASA (Model-Aware Skill Alignment) is the fix for that mismatch — a two-stage pipeline that rewrites skill text for each target backbone without touching its weights. Stage one runs a hierarchical skill evolution loop: general, cross-task skills get refined by hill climbing, while task-specific skills go through UCB-driven tree search that branches and prunes candidate rewrites against environment feedback. Both stages condition on a model card — a structured profile of the backbone's architecture, training provenance, and known behavioral strengths and weaknesses. Stage two distills the search trajectories into a lightweight skill rewriter that reproduces the adaptation in a single forward pass — no further search needed at deployment.

Tested across three interactive environments — ALFWorld household tasks, WebShop shopping, and a search-augmented QA benchmark — MASA beats the strongest baseline for every backbone, with gains up to 25.8 points on ALFWorld's 8B model. The sharper failure mode shows up on WebShop, where larger models paradoxically underperform smaller ones without skill support: Qwen3-14B manages just 2.8% success, part of a pattern the paper attributes to larger models' more verbose chain-of-thought generation. MASA lifts that to 29.2% — a large recovery, though still short of the 32B backbone's 34.6% under MASA. The gains carry over to unseen tasks and environments without rerunning search — the distilled rewriter even outperforms DeepSeek-V4-Pro, the much larger teacher model used to generate its own training trajectories, at a fraction of the inference cost, though the paper doesn't put a number on that gap.