Flat Skill Invocation Craters From 48 to 17 as the Pool Hits 200,000

Notes on Organizing, Orchestrating, and Benchmarking Agent Skills at Ecosystem Scale (arXiv:2603.02176) — Hao Li, Chunjiang Mu, Jianhao Chen, Siyue Ren, Zhiyao Cui, Yiqun Zhang, Lei Bai, Shuyue Hu · March 2026

Note published July 23, 2026 · written by SkillFed’s research pipeline from the paper above · how these notes are made

AgentSkillOS tackles a scale problem: as of late February 2026, the Claude skill ecosystem held more than 280,000 public skills, built almost entirely by decentralized third-party authors, with no structured way for a user or agent to browse that space. The fix runs in two stages. Offline, a capability tree organizes the pool through recursive LLM categorization — each node splits into child categories once its skill count exceeds a per-node capacity, collapsing even a 200,000-skill ecosystem into a shallow, browsable hierarchy. Past a size threshold, only the top-installed skills stay in this active tree (10,000 of 200,000 at the largest scale tested); the rest sit in a dormant, embedding-searchable index. At task time, the system walks the tree to shortlist candidates, prunes down to the top eight, and compiles them into a DAG-based orchestration plan under one of three named strategies (Quality-First, Efficiency-First, Simplicity-First) before executing skills in dependency order. Testing this against the obvious alternative — handing an agent the same skills unstructured — required a benchmark built for the purpose: 30 tasks spanning five artifact categories (data computation, document creation, motion video, visual design, web interaction), graded by pairwise LLM judging distilled into a Bradley-Terry model score.

Tested at three ecosystem sizes — 200, 1,000, and 200,000 skills — the tree-plus-DAG system holds the top rank at every scale. The same skills, handed to Claude Code as one unstructured pool (flat invocation), get worse as the pool grows: rescaled score falls from 48.1 at 1,000 skills to 17.2 at 200,000, as a growing share of skills becomes effectively invisible to unstructured selection. The sharper result is the ablation. Even when the flat agent is handed the exact oracle skill set for a task — no retrieval problem left to solve — it still trails the DAG-organized version working from the identical skills. Tree-based retrieval alone closes most of the distance to that oracle selection, and the residual gap shrinks further at 200,000 skills: hierarchical retrieval degrades gracefully where flat invocation does not.