A fine-tuned 8B retriever hits 83 NDCG@10 — a 12B off-the-shelf model manages 55

Notes on SkillRet: A Large-Scale Benchmark for Skill Retrieval in LLM Agents (arXiv:2605.05726) — H. Cho, Ryan Kang, Youngeun Kim · May 2026

Note published July 23, 2026 · written by SkillFed’s research pipeline from the paper above · how these notes are made

AI-assisted notes · reviewed by SkillFed Agentic benchmarks

SkillRet turns skill retrieval into its own measurable problem, separate from end-to-end agent benchmarks. The corpus starts at 22,795 public agent skills scraped from claude-plugins.dev, then narrows through five filtering passes — language, license (MIT/Apache-2.0 only), two rounds of deduplication — down to 17,810 skills, each tagged into a two-level taxonomy of 6 major and 18 sub-categories. These are long documents, not short tool blurbs: median length is 1,583 tokens, an order of magnitude past typical tool-retrieval benchmarks. Training queries (63,259) and evaluation queries (4,997) were generated separately over disjoint skill pools — Qwen3.5-122B-A10B for training, Claude Opus 4.6 for evaluation — then filtered for skill-name leakage and checked by both an LLM panel and human annotators. Just over half the evaluation queries name two or three skills at once, not one.

Across 18 embedding models plus rerankers, retrieval quality stays well below where a production agent would want it. The best off-the-shelf model (harrier-oss-v1-0.6b) reaches 66.55 NDCG@10; the best prior fine-tuned skill retriever, SkillRouter, reaches 70.38. Parameter count alone doesn't help: a 12B general embedding model scores just 55.38, trailing several 0.6B and 8B competitors. Fine-tuning on SkillRet's own training split is what actually moves the needle — SkillRet-Embedding-8B reaches 83.45 NDCG@10, a 16.9-point gain over the best off-the-shelf model and 13.1 over SkillRouter. A sentence-masking analysis pins down why: on single-skill queries, deleting the one sentence that names the needed capability drops the fine-tuned model's score by 29.2%, versus 23.3% for its untrained base. The fine-tuned model has learned to find a needle in a long, scenario-heavy query rather than average over the whole thing. And a model's rank on the general MTEB Retrieval benchmark only moderately predicts its SkillRet rank (Spearman ρ=0.71) — enough inversions that leaderboard position isn't a reliable proxy for skill-finding ability.

Key numbers

Curated skill corpus	17,810 skills (from 22,795 crawled)
Best off-the-shelf retriever	66.55 NDCG@10
Best prior skill-retrieval model	70.38 NDCG@10
SkillRet-Embedding-8B (fine-tuned)	83.45 NDCG@10
MTEB rank vs. SkillRet rank	Spearman ρ=0.71, frequent inversions

What this means for skill authors

The methodological split matters more than it looks. Recent skill benchmarks like SkillsBench and SWE-Skills-Bench ask whether a retrieved skill helps once used; SkillRet asks whether the right skill gets retrieved in the first place. Conflating the two hides where an agent pipeline actually breaks. As skill libraries grow past what fits in a context window or a user's memory of names, retrieval — not authoring, not orchestration — becomes the first point of failure, and this paper gives that failure point an actual measuring stick at real scale.

For skill authors, the sentence-erasure result is a writing lesson as much as a modeling one: a skill only surfaces when some part of its own text lines up with the one or two sentences in a request that actually carry intent. Burying the capability description under boilerplate or setup instructions makes a skill structurally harder to find, independent of how good the retriever is. The category breakdown backs this up — Information Retrieval and AI Agents skills lag Software Engineering and Data & ML by double digits even under the best fine-tuned model, which reads less like an inherent difficulty gap and more like these categories' descriptions carry weaker, more diffuse signal.

The MTEB-doesn't-transfer finding is the most useful data point for anyone building skill infrastructure rather than benchmarking it: picking a retriever off a general embedding leaderboard is not a substitute for evaluating it on the actual skill corpus. That's the argument for treating skill discovery as its own vetted, domain-tuned layer — closer to how SkillFed's catalog matches an abstract task description against curated, tagged skills — rather than pointing a generic off-the-shelf embedding search at a folder of SKILL.md files and hoping semantic similarity does the rest.

Skills related to this research

ai-bot-log-audit Track and interpret how AI agents interact with your infrastructure by examining server logs for bot activity patterns. Gain actionable insights into crawl frequency, request timing, and resource usage to refine your content strategy and improve bot-to-human traffic balance. Essential for teams managing high-volume AI agent access.★ 140 wiki-retrieve wiki-retrieve upgrades vault search from page-level to chunk-level granularity, combining contextual prefixes, BM25 sparse retrieval, and dense reranking to reduce retrieval failures. Setup is opt-in with built-in privacy gates: run fully on-machine with synthetic prefixes, or opt into Anthropic API or claude CLI tiers for better context generation.★ 10,007 codspeed-optimize This skill acts as an autonomous performance engineer, using CodSpeed benchmarks and flamegraph analysis to iteratively optimize code. It measures baseline performance, identifies bottlenecks through flame graphs, applies targeted changes, and validates improvements across simulation and walltime modes. The skill handles the full optimization loop—from establishing baselines to comparing runs and reporting gains—stopping only when significant improvements plateau or the user decides to halt.★ 235 codspeed-setup-harness This skill guides you through configuring benchmarks and integrating CodSpeed into your project. It covers language-specific harnesses (divan, criterion, pytest-benchmark, vitest, go test, Google Benchmark) and a universal exec mode for any language, helping you identify performance-critical code and establish repeatable performance tracking.★ 235

Related notes

References

H. Cho, R. Kang, Y. Kim, "SkillRet: A Large-Scale Benchmark for Skill Retrieval in LLM Agents," arXiv:2605.05726 (2026).
Y. Zheng et al., "SkillRouter: Skill Routing for LLM Agents at Scale," arXiv:2603.22455 (2026).
Z. Shi et al., "Retrieval Models Aren't Tool-Savvy: Benchmarking Tool Retrieval for Large Language Models" (ToolRet), arXiv:2503.01763 (2025).
X. Li et al., "SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks," arXiv:2602.12670 (2026).