A 96,401-skill curated corpus lifts agent pass rates +7.5pp — until coverage runs out

Notes on SkillCorpus: Consolidating and Evaluating the Open Skill Ecosystem for Real-World LLM Agents (arXiv:2607.15557) — Yanze Wang, Pengfei Yao, Tianyi Sun, Chuanrui Hu, Yan Xiao, Yunyun Han, Jun Sun, Yafeng Deng · July 2026

Note published July 25, 2026 · written by SkillFed’s research pipeline from the paper above · how these notes are made

SkillCorpus turns the sprawling public SKILL.md ecosystem into one deployable, licence-clean corpus. A six-stage pipeline parses, deduplicates, and quality-scores roughly 821,000 crawled skill files down to 96,401 survivors: an exact-match tier collapses 169,465 duplicates outright, a semantic tier merges near-duplicates above a cosine-similarity threshold and sends borderline pairs to an LLM judge, and a final safety hard-gate plus OSI-licence filter removes skills flagged for prompt injection, command injection, unsafe execution, or an unclear licence. Every surviving skill gets one of 16 task-class labels and three independent quality scores — utility, robustness, and safety — instead of one blended number, and the corpus ships with a fine-tuned retrieval-and-selection stack: an embedding recall model, a reranker, and an LLM gate that reads each candidate's full body before injecting up to two skills into an agent's prompt.

Plugging that corpus into live agent runs produces gains across every setup tested — three benchmarks, two independently-built open-source harnesses, two open backbones, and a Claude Opus 4.7 spot-check — but the size of the gain is far from uniform. SkillsBench pass rates rise 7.5 points on average across the four harness/backbone pairings (as much as 13.4 points on the strongest pairing, and +8.0 points on the Opus 4.7 check); GDPVal and QwenClawBench, which start from much higher no-skill baselines, move by roughly 1.5 and 2.8 points. Two variables explain most of the spread. One is how well the corpus actually covers a task's domain: binning tasks by retrieval-match quality, the mean gain climbs from +2.2 points in the worst-covered bin to +25.1 points in the best-covered one. The other is which harness executes the skill: handed the identical retrieved skill, one open-source harness gained 13.4 points on a benchmark while the other gained only 5.8, because one ran an execute-verify-fix loop to completion and the other stopped after drafting code it never ran.