Skill Routing Loses 37-44 Points When It Can't See the Body

Notes on SkillRouter: Skill Routing for LLM Agents at Scale (arXiv:2603.22455) — Yanzhao Zheng, Zhentao Zhang, Chao Ma, Yuanqiang Yu, Jihuai Zhu, Yongliang Wu, Tianze Xu, Baohua Dong, Hangcheng Zhu, Ruohui Huang, Gang Yu · March 2026

Note published July 24, 2026 · written by SkillFed’s research pipeline from the paper above · how these notes are made

LLM agent skill catalogs are heading into the same problem search ran into decades ago: too many candidates to hand a model all at once. The standard fix is progressive disclosure — a router picks a skill from just its name and description, and only the winner's full implementation body ever enters the agent's context. Zheng et al. test whether that shortcut holds once a registry gets large and crowded with near-duplicates. They built an approximately 80,000-skill benchmark derived from SkillsBench, split into an Easy tier and a Hard tier salted with 780 LLM-generated distractor skills engineered to look right while doing the wrong thing. Then they ran the same skill routing test twice across both encoder-only retrieval and retrieve-and-rerank baselines — once with the skill body visible to the router, once with it hidden.

Hiding the body is expensive: Hit@1 routing accuracy drops 37 to 44 percentage points across every baseline tested, dense or reranking. Two controls rule out the boring explanation that the router just needed more tokens, any tokens. Feeding it an LLM-distilled, body-derived description recovers much of the gap — but still trails direct body access by 7 to 21 points. Train a metadata-only encoder on the exact same data and negatives as the all-field version, and a 14.0-point shortfall remains anyway. The missing signal lives in the literal body content, not in text volume. SkillRouter, the paper's own system, acts directly on that finding: a compact 1.2B retrieve-and-rerank pipeline — a 0.6B bi-encoder retriever plus a 0.6B cross-encoder reranker, both reading every skill field — hits 74.0% Hit@1, ahead of a 16B baseline's 68.0%, while running 13 times smaller and 5.8 times faster. Two training choices do most of that work: false-negative filtering that strips out near-duplicate skills wrongly mined as negatives, and a listwise reranking loss in place of a pointwise one (+30.7 points). The gains carry through to real task success, too, across four coding agents — Kimi-K2.5, GLM-5, Claude Sonnet 4.6, and Claude Opus 4.6. The more capable the agent, the bigger the payoff from better routing.