Retrieving the right skill lifts coding Pass@1 78%, reaching 84% of oracle-level performance

Notes on SkillFlow: Scalable and Efficient Agent Skill Retrieval System (arXiv:2504.06188) — Fangzhou Li, Pagkratios Tagkopoulos, Ilias Tagkopoulos · April 2025

Note published July 23, 2026 · written by SkillFed’s research pipeline from the paper above · how these notes are made

AI-assisted notes · reviewed by SkillFed Skill retrieval

SkillFlow treats skill selection as a search problem, not a context-stuffing problem. Given a task, it runs dense retrieval over roughly 36,000 community-contributed SKILL.md files scraped from GitHub, pulling back about 1,000 candidates with a bi-encoder. Two rounds of cross-encoder reranking narrow that further: a shallow pass on truncated text cuts to around 100 candidates, then a deep pass on full content cuts to about 10, before an LLM makes the final call and keeps five skills or fewer. The pipeline is evaluated on two coding benchmarks: SkillsBench, where 87 tasks come paired with 229 known oracle-matched skills, and Terminal-Bench, 89 tasks with no matched-skill ground truth at all.

On SkillsBench, skills that SkillFlow retrieves raise agent Pass@1 from 9.2% to 16.4% -- a 78% relative gain that reaches 84% of the oracle ceiling, the score an agent gets when handed the ideal skill directly. Terminal-Bench tells a different story: agents pull a retrieved skill into 70.1% of their runs, yet Pass@1 doesn't move. That gap traces to what's actually inside the skills, not how well they're found: oracle skills carry a higher share of fenced code (median code fraction 0.39 vs. 0.24) and are more than twice as likely to bundle an executable script (33.2% vs. 13.4%). Retrieval surfaces what exists in the corpus; it can't manufacture domain coverage the corpus never had.

Key numbers

SkillsBench Pass@1: no-skills baseline vs. SkillFlow-retrieved	9.2% → 16.4% (+78.3%)
Share of oracle ceiling reached	84.1%
Terminal-Bench skill adoption with no Pass@1 gain	70.1% use rate, no improvement
Bundled executable scripts: oracle vs. community skills	33.2% vs. 13.4%
Community skill corpus indexed from GitHub	~35,866 SKILL.md files

What this means for skill authors

This is the clearest evidence yet that retrieval, done properly, earns its keep for skill-augmented agents -- a 78% relative jump in task success is not a rounding error. But the size of that jump is set by the library, not the pipeline. Length and inline code density don't separate oracle skills from average community ones -- what does is whether the skill actually ships a runnable artifact.

That's a direct instruction for anyone authoring a skill: a SKILL.md that only describes a procedure in prose is competing against skills that ship a working script, and the data says the script wins. For a vetted catalog, the retrieval funnel itself -- dense search, then two narrowing reranking passes, then an LLM gate -- is close to solved engineering; the leverage sits upstream, in what gets indexed before an agent ever queries it. A dual-tool-scanned, quality-gated catalog is effectively pre-selecting for the same code-density and bundled-artifact signal this paper had to discover after the fact by scoring 36,000 scraped skills one by one.

One more flag for anyone building or ranking a skill marketplace: usage isn't validation. Agents pulled Terminal-Bench skills into 70% of their runs with zero net benefit, meaning adoption telemetry alone would have called this pipeline a win. Any ranking signal built from how often a skill gets picked needs a task-outcome check behind it -- otherwise the metric rewards what gets grabbed, not what actually helps.

Skills related to this research

wiki-retrieve wiki-retrieve upgrades vault search from page-level to chunk-level granularity, combining contextual prefixes, BM25 sparse retrieval, and dense reranking to reduce retrieval failures. Setup is opt-in with built-in privacy gates: run fully on-machine with synthetic prefixes, or opt into Anthropic API or claude CLI tiers for better context generation.★ 10,007 baoyu-xhs-images Transform text into visually striking image card sequences designed for social platforms. Choose from 12 visual styles, 8 layout options, and 3 color palettes to match your brand, then watch your content break down into 1–10 shareable cards. Built for creators who want polished, on-brand infographics without the design work.★ 24,241 nextjs-react-redux-typescript-cursor-rules This skill delivers a complete ruleset for building maintainable Next.js and React applications with Redux Toolkit and TypeScript. It covers code style conventions, component patterns, performance optimization, state management strategies, and accessibility standards grounded in SOLID principles and functional programming approaches.★ 202 Baoyu Xhs Images This skill breaks down content into engaging infographic series tailored for Xiaohongshu, offering 10 distinct visual styles—from cute and fresh to notion and study-notes—combined with 8 layout options like dense, comparison, and mindmap. Choose your aesthetic and information structure, or let it auto-select based on content signals. Generates 1-10 optimized images per session with organized file output.★ 331

Related notes

References

Li, F., Tagkopoulos, P., & Tagkopoulos, I. (2025). SkillFlow: Scalable and Efficient Agent Skill Retrieval System. arXiv:2504.06188.
Li, X., Chen, W., Liu, Y., et al. (2026). SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks. arXiv:2602.12670.
Merrill, M. A., Shaw, A. G., Carlini, N., et al. (2026). Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command-Line Interfaces. ICLR.
Karpukhin, V., Oğuz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., & Yih, W. (2020). Dense Passage Retrieval for Open-Domain Question Answering. EMNLP.
Lewis, P., et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. NeurIPS.