Only 0.1 points separate agents that need a skill from ones that don't

Notes on Skill Retrieval Augmentation for Agentic AI (arXiv:2604.24594) — Weihang Su, Jianming Long, Qingyao Ai, Yichen Tang, Changyue Wang, Yiteng Tu, Yiqun Liu · April 2026

Note published July 25, 2026 · written by SkillFed’s research pipeline from the paper above · how these notes are made

AI-assisted notes · reviewed by SkillFed Agentic benchmarks Bridge: benchmarks × retrieval

Cramming every skill's SKILL.md into context stops scaling long before a library gets large: context budgets fill up, and the calling model gets measurably worse at picking the right entry out of the pile. Skill Retrieval Augmentation (SRA) splits that into three separable stages instead of one context-stuffing problem — retrieve candidates from a large external corpus, decide which of them actually make it into the active context (skill incorporation), then apply whatever got loaded to solve the task — and SRA-Bench grades each stage on its own rather than scoring pass/fail on the end result. The benchmark draws 5,400 test instances from six existing datasets spanning theorem proving, formal logic, tool-use QA, medical calculation, competition math, and code generation, each hand-annotated against one or more of 636 gold skills. Those gold skills sit buried inside 25,626 web-scraped distractor skills, producing a 26,262-entry corpus where only 2.4% of anything is actually relevant to a given query. Eight models — six open-weight (Qwen3-4B/32B/235B, Llama-3.1-8B, Llama-3.3-70B, Mistral-Small-3.1-24B) plus frontier GLM-5.1 and GPT-5.4 — ran across BM25, dense, and reranking retrieval, and three skill-use strategies: full injection of the single top-ranked result, LLM-driven selection from a top-50 shortlist, and an OpenClaw-style progressive disclosure setup where the agent pulls full skill content on demand instead of all at once.

Retrieval-based augmentation earns its keep: letting a model pick from its own top-50 retrieved candidates lifted end-task accuracy 7.2 to 14.5 percentage points over a skill-free baseline, consistent across all six open models tested. The sharper finding sits one stage downstream, in incorporation. Overall skill-loading rates range from 11.1% (Llama-3.3-70B) to 72.1% (Llama-3.1-8B) under an identical corpus and retriever — a spread that tracks model identity, not scale or task demand, since the 70B model loads far less often than the 8B one. Most models barely adjust their loading behavior based on whether the correct skill is actually sitting among their retrieved candidates: Llama-3.3-70B's overall loading rate runs 4.2 points lower when the gold skill is present in the top-50 than when it isn't. Only the frontier models tested, GLM-5.1 and GPT-5.4, show a real split between gold-present and gold-absent cases (+35.4pp and +33.0pp respectively) — and even they don't close the deeper gap. Across the benchmark, agents load skills at essentially the same rate whether the task is one they can already solve unaided or one they demonstrably cannot: a 0.1-percentage-point difference between the two conditions, about as close to zero need-aware skill loading as a benchmark result gets.

Key numbers

Skill corpus scale	26,262 skills, only 2.4% gold (636)
Accuracy gain, LLM-selected retrieval vs. skill-free	+7.2 to +14.5 pp across six models
Skill-loading rate spread across models	11.1% (Llama-3.3-70B) to 72.1% (Llama-3.1-8B)
Relevance-blindness, worst case	Llama-3.3-70B loads 4.2 pp LESS when gold skill is retrievable
Need-awareness gap, aggregated	+0.1 pp (solvable vs. unsolvable tasks)

What this means for skill authors

The relevance-awareness numbers are the more actionable ones for anyone shipping into a shared skill catalog: retrieval quality is necessary and clearly not sufficient. A perfectly-ranked top-50 with the right skill sitting at position one still gets ignored by most tested models nearly as often as it gets loaded. That reframes what "discoverable" has to mean for a skill author — writing metadata that ranks well only solves half the problem. The other half is surviving whatever heuristic the calling model uses to decide a candidate is worth opening at all, and right now that heuristic looks like a per-model habit more than a reasoned judgment call.

The near-zero need-awareness gap is the harder problem, and it isn't one a skill author can fix from their side of the catalog. If a model can't reliably tell "I need outside help here" from "I've got this," it will load skills it doesn't need and skip the ones it does, regardless of how good the corpus or the retriever is. That argues for treating skill invocation as an unreliable classifier worth gating explicitly, rather than a judgment call left entirely inside the base model. Dual-tool scanning and a vetted catalog narrow the discovery half of this problem — putting the right skill within reach — but the incorporation decision still happens inside the calling model, and this paper is fairly direct evidence that the decision isn't trustworthy on its own yet.

The scale motivation isn't hypothetical, either: the paper cites skill marketplaces already listing upward of a million distinct entries at the time of writing. Enumeration-in-context was always going to break at some size; SRA is a bet on what has to replace it, and SRA-Bench is the first real attempt to measure where that replacement currently falls short.

Skills related to this research

wiki-retrieve wiki-retrieve upgrades vault search from page-level to chunk-level granularity, combining contextual prefixes, BM25 sparse retrieval, and dense reranking to reduce retrieval failures. Setup is opt-in with built-in privacy gates: run fully on-machine with synthetic prefixes, or opt into Anthropic API or claude CLI tiers for better context generation.★ 10,007 postfx-effects postfx-effects delivers a collection of post-processing filters designed to enhance 3D scene rendering with cinematic quality. Apply bloom, color grading, depth-of-field, and other visual effects to transform raw renders into polished final output. Integrate seamlessly into game engines and real-time graphics pipelines.★ 8 creator-content-auditor This skill automates the review process for creator-submitted content, comparing each piece against your established brand guidelines and campaign parameters. It helps marketing teams catch misalignments early, maintain consistent brand voice across influencer partnerships, and streamline approval workflows before content goes live.★ 2,469 Context7 Context7 fetches the latest library documentation to give you accurate, up-to-date API information and code examples. It eliminates hallucinations from outdated training data by retrieving version-specific details when you ask about library usage, request examples, or explicitly invoke it.★ 3

Related notes

References

Su, Long, Ai, Tang, Wang, Tu & Liu, "Skill Retrieval Augmentation for Agentic AI" (arXiv:2604.24594, 2026)
Lewis et al., "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks" (NeurIPS 2020)
Wang et al., "Voyager: An Open-Ended Embodied Agent with Large Language Models" (arXiv:2305.16291, 2023)
Schick et al., "Toolformer: Language Models Can Teach Themselves to Use Tools" (NeurIPS 2023)
Zhuang, Yu, Wang, Sun & Zhang, "ToolQA: A Dataset for LLM Question Answering with External Tools" (arXiv, 2023)