Retrieving the right skill lifts coding Pass@1 78%, reaching 84% of oracle-level performance

Notes on SkillFlow: Scalable and Efficient Agent Skill Retrieval System (arXiv:2504.06188) — Fangzhou Li, Pagkratios Tagkopoulos, Ilias Tagkopoulos · April 2025

Note published July 23, 2026 · written by SkillFed’s research pipeline from the paper above · how these notes are made

SkillFlow treats skill selection as a search problem, not a context-stuffing problem. Given a task, it runs dense retrieval over roughly 36,000 community-contributed SKILL.md files scraped from GitHub, pulling back about 1,000 candidates with a bi-encoder. Two rounds of cross-encoder reranking narrow that further: a shallow pass on truncated text cuts to around 100 candidates, then a deep pass on full content cuts to about 10, before an LLM makes the final call and keeps five skills or fewer. The pipeline is evaluated on two coding benchmarks: SkillsBench, where 87 tasks come paired with 229 known oracle-matched skills, and Terminal-Bench, 89 tasks with no matched-skill ground truth at all.

On SkillsBench, skills that SkillFlow retrieves raise agent Pass@1 from 9.2% to 16.4% -- a 78% relative gain that reaches 84% of the oracle ceiling, the score an agent gets when handed the ideal skill directly. Terminal-Bench tells a different story: agents pull a retrieved skill into 70.1% of their runs, yet Pass@1 doesn't move. That gap traces to what's actually inside the skills, not how well they're found: oracle skills carry a higher share of fenced code (median code fraction 0.39 vs. 0.24) and are more than twice as likely to bundle an executable script (33.2% vs. 13.4%). Retrieval surfaces what exists in the corpus; it can't manufacture domain coverage the corpus never had.