GitHub-mined skills teach 40% better, but 26% ship vulnerabilities

Notes on Automating Skill Acquisition through Large-Scale Mining of Open-Source Agentic Repositories: A Framework for Multi-Agent Procedural Knowledge Extraction (arXiv:2603.11808) — Shuzhen Bi, Mengsong Wu, Hao Hao, Keqian Li, Wentao Liu, Siyu Song, Hongbo Zhao, Aimin Zhou · March 2026

Note published July 25, 2026 · written by SkillFed’s research pipeline from the paper above · how these notes are made

AI-assisted notes · reviewed by SkillFed Skill security Bridge: security × retrieval

The pipeline mines open-source agentic repositories in three stages: structural analysis of a repo's layout, dense retrieval to find reusable procedural capabilities buried in the code, and translation of what it finds into the standardized SKILL.md format. Run against two Manim-based educational video generators — TheoremExplainAgent's planner-coder pair and Code2Video's three-agent planner-coder-critic setup — it produces two working skills: a theorem-walkthrough animator and a visual-layout critic. Neither ships without clearing a four-gate check first: static code analysis, semantic prompt injection screening, sandboxed behavioral testing, and permission validation against an allowed-tools manifest.

That screen matters. Run across the full mined pool, it flags 26.1% of artifacts with vulnerabilities — the paper's own basis for treating scraped code as needing graduated trust tiers, not blanket trust. The capability side looks better: extracted skills delivered a 40% gain in knowledge-transfer efficiency on downstream educational tasks, held pedagogical quality roughly level with human-crafted tutorials, and — composed through an ontological structure the authors call SkillNet — cut execution steps by 30% versus running skills independently.

Key numbers

Knowledge-transfer efficiency gain from mined skills	40%
Mined artifacts flagged with vulnerabilities	26.1%
Execution-step reduction via SkillNet composition	30%
Skills navigable via progressive disclosure	10,000+
Cost reduction vs. retraining a model	2-3 orders of magnitude

What this means for skill authors

GitHub turns out to be a legitimate skill-mining ground, but the paper's own security numbers double as the counter-argument: a roughly one-in-four contamination rate among scraped artifacts isn't noise — it's the reason extraction pipelines need a verification gate built in, not bolted on after. Mining procedural knowledge and vetting it are two different jobs, the same split behind SkillFed's dual-tool scanning before anything reaches a vetted catalog.

For skill authors: provenance from a well-known agentic repo signals capability quality, not safety — those get certified on separate axes. For skill users: treating "extracted from a real production agent" as a trust shortcut is exactly the mistake this data warns against — a quarter of that population still failed the check.

SKILL.md's three-level structure is token-budgeted for exactly this scaling problem — a 30-100 token metadata card, a 200-5,000 token procedure body, unbounded reference material below that — which is how an agent stays aware of a claimed 10,000-plus skills without loading all of them into context: progressive disclosure. Add the 30% step reduction from composing skills via SkillNet, and the implied end-state isn't a flat plugin list. It's a searchable, composable library — the same bet a federated, vetted catalog is making, qualitatively.

Skills related to this research

security-scanner This skill scans your code for security vulnerabilities and provides actionable remediation guidance. It helps development teams quickly identify and address potential risks before they reach production, streamlining the security review process.★ 84 wiki-retrieve wiki-retrieve upgrades vault search from page-level to chunk-level granularity, combining contextual prefixes, BM25 sparse retrieval, and dense reranking to reduce retrieval failures. Setup is opt-in with built-in privacy gates: run fully on-machine with synthetic prefixes, or opt into Anthropic API or claude CLI tiers for better context generation.★ 10,007 pi-librarian Pi-librarian is a GitHub research subagent that searches external repositories, locates relevant files, and generates cited reports to inform your implementation decisions. It outputs a detailed research report and actionable summary, helping you compare patterns and gather concrete evidence from GitHub before coding.★ 406 creator-content-auditor This skill automates the review process for creator-submitted content, comparing each piece against your established brand guidelines and campaign parameters. It helps marketing teams catch misalignments early, maintain consistent brand voice across influencer partnerships, and streamline approval workflows before content goes live.★ 2,469

Related notes

References

Bi, Wu, Hao, Li, Liu, Song, Zhao, Zhou — "Automating Skill Acquisition through Large-Scale Mining of Open-Source Agentic Repositories" (arXiv:2603.11808, March 2026)
TheoremExplainAgent — planner/coding-agent system for Manim-based theorem explanation videos, one of the two mined source repositories
Code2Video — tri-agent (planner/coder/critic) framework for code-driven educational video generation, the second mined source repository