Forced to search a 34k-skill library instead of being handed the right one, agents' pass rate falls from 55.4% to 38.4% -- a hair above the 35.4% no-skill floor

Notes on How Well Do Agentic Skills Work in the Wild: Benchmarking LLM Skill Usage in Realistic Settings (arXiv:2604.04323) — Yujian Liu, Jiabao Ji, Li An, T. Jaakkola, Yang Zhang, Shiyu Chang · April 2026

Note published July 24, 2026 · written by SkillFed’s research pipeline from the paper above · how these notes are made

Liu et al. build a 34,198-skill test bed by pulling metadata from two community skill hubs, skillhub.club and skills.sh, then downloading the full SKILL.md folders from their GitHub sources -- keeping only permissively licensed, well-formed, de-duplicated entries. Against that collection they run three models, Claude Opus 4.6, Kimi K2.5, and Qwen3.5-397B-A17B, through six settings of increasing realism on SkillsBench's 84 tasks (three runs each): skills hand-picked and forced into context, hand-picked without forcing, hand-picked with distractor skills mixed in, retrieved by the agent itself from the full 34k with the right skill still in the pool, retrieved with it removed from the pool, and a no-skill floor. A second track scores five retrieval strategies by recall@5 -- a single-shot dense-embedding lookup versus four agentic variants where the model itself issues search queries and judges what comes back -- and tests two ways to fix up whatever skill gets retrieved: query-specific refinement, which adapts the skill to the task at hand, and query-agnostic refinement, a one-time edit made with no particular task in view.

The benefit shrinks every time the setting gets more realistic. Claude's SkillsBench pass rate runs 55.4% with skills hand-picked and forced into context, 51.2% just hand-picked, 43.5% once distractor skills are mixed in, 40.1% once the agent has to retrieve rather than being handed anything, and 38.4% once the right skill isn't even in the retrieval pool -- a hair above the 35.4% no-skill baseline. Kimi and Qwen trace the same slope down toward their own baselines. Underneath that curve sits a selection failure, not just a retrieval one: even when the right skill sits in context, Claude loads all of it in only 49% of runs, falling to 31% once distractors are added. Retrieval quality varies just as sharply -- plain dense search over the metadata index catches the right skill only 47.0% of the time at five results, while giving the agent search tools and full skill content to iterate against pushes that to 65.5%. What actually recovers lost ground is adapting the retrieved skill to the task in front of it: query-specific refinement lifts Claude's retrieved score from 40.1% to 48.2% on SkillsBench and carries its Terminal-Bench 2.0 pass rate from a 57.7% no-skill baseline through 61.4% on retrieval alone to 65.5%. Query-agnostic refinement, the cheaper one-time edit, barely moves Claude (40.1% to 42.0%) and slightly hurts Qwen (26.7% to 26.2%).