Checking the repo, not just the SKILL.md, cuts flagged 'malicious' skills from 46.8% to 0.52%

Notes on Malicious Or Not: Adding Repository Context to Agent Skill Classification (arXiv:2603.16572) — Florian Holzbauer, David Schmidt, G. Gegenhuber, S. Schrittwieser, Johanna Ullrich · March 2026

Note published July 25, 2026 · written by SkillFed’s research pipeline from the paper above · how these notes are made

AI-assisted notes · reviewed by SkillFed Skill security

Researchers crawled 238,180 unique agent skills from ClawHub, Skills.sh, SkillDirectory, and GitHub — the largest cross-platform census of the skill ecosystem so far. Every skill passed through three lenses: each marketplace's own scanner, an independent Cisco skill scanner, and the authors' own LLM-based classifier. For skills flagged high-risk by both an external scanner and their model, the team went a step further and checked the skill against the GitHub repository it actually ships from — README, code, commit history — rather than judging the SKILL.md file in isolation.

Marketplace-reported malicious rates swing from 46.8% on ClawHub down to 6.0% on SkillDirectory, and the five scanners tested barely agree with each other: among the 27,111 skills all five tools analyzed in common, only 33 (0.12%) got flagged by every one of them. Aggregating flags at the repository level roughly doubles the apparent risk, too — 19.3% of individual skills get flagged versus 45.9% of the repositories hosting them, since one bad skill is enough to tag an entire repo as malicious. Once flagged skills are checked against their home repository, the suspicious set nearly disappears: of 2,887 scanner-flagged skill-repository pairs fully evaluated, only 15 (0.52%) still looked malicious once repository alignment and maturity were factored in, and manual reviewers who checked a sample of flagged repos found nearly all of them benign. The same repository-level lens surfaced real risk elsewhere — 121 skills route through seven abandoned, hijackable GitHub repositories, and a dozen live API credentials, including NVIDIA, ElevenLabs, and MongoDB tokens, sit exposed inside published skills.

Key numbers

Unique skills analyzed across 4 platforms	238,180
ClawHub-reported malicious rate	46.8%
Flagged skills still suspicious after repo context	0.52% (15 of 2,887)
Agreement across all 5 scanners tested	0.12% (33 skills)
Skills exposed to repository hijacking	121 (via 7 abandoned repos)

What this means for skill authors

The headline finding isn't that skills are safe — it's that scanning a SKILL.md file in isolation is a poor proxy for risk, and publishing a raw scanner fail rate trains users toward alarm fatigue. A 46.8% malicious rate reads like a crisis. Look at the repository a skill actually ships from, and the number that survives scrutiny drops to about one in two hundred. That gap is the whole finding, and it should change how anyone reads a marketplace's published security badge.

It also reframes single-scanner triage as noisy, not just imperfect: five tools looking at the same skill population agreed on almost nothing, with only 33 skills out of thousands drawing a flag from all five. Treat scanner output as one signal to corroborate against description, code, and repository context — not a verdict a catalog publishes on its own. That's the kind of dual-tool, context-aware vetting this paper's own methodology effectively argues for.

For skill authors, the actionable risk is the boring supply-chain kind: don't leave live API credentials sitting in a public skill repo, and don't publish a skill whose install path depends on a GitHub repository you might abandon. This paper found 121 skills already exposed to takeover through exactly that path, with the most-downloaded hijackable skill logged past two thousand downloads before anyone noticed. It's the same lesson package registries and app stores learned years ago, wearing a new SKILL.md wrapper.

Skills related to this research

pi-librarian Pi-librarian is a GitHub research subagent that searches external repositories, locates relevant files, and generates cited reports to inform your implementation decisions. It outputs a detailed research report and actionable summary, helping you compare patterns and gather concrete evidence from GitHub before coding.★ 406 10 Music Video This skill transforms your music into precise visual prompts for Seedance 2.0 on Higgsfield, a platform that synchronizes generated video to your uploaded audio track. Learn to reference specific beats, describe genre-matched visuals, and apply proven hooks—from particle explosions on drops to color shifts on harmonic changes—that lock your visuals to every moment of your song.★ 712 add-generation-support Integrate an existing ecosystem into the generation system by wiring it into the form, orchestrator, and router. This skill creates the necessary graph and handler files, updates basemodel constants, and configures workflow routing—use it after add-ecosystem when you need an ecosystem to appear in generation UI.★ 7,201 System Architecture System Architecture guides structural decisions for Python/React full-stack projects during the design phase. It covers backend layering (routes, services, repositories, models), frontend component hierarchy, state management patterns, and cross-cutting concerns like authentication and error handling. Use it to plan module boundaries, evaluate technology trade-offs, and document decisions via Architecture Decision Records.★ 8

Related notes

References

F. Holzbauer, D. Schmidt, G. K. Gegenhuber, S. Schrittwieser, J. Ullrich, "Context Matters: Repository-Aware Security Analysis of the Agent Skill Ecosystem," arXiv:2603.16572 (2026).
L. Beurer-Kellner, A. Kudrinskii, M. Milanta, K. B. Nielsen, H. Sarkar, L. Tal, "Snyk Finds Prompt Injection in 36%, 1467 Malicious Payloads in a ToxicSkills Study of Agent Skills Supply Chain Compromise" (2026).
Y. Liu, W. Wang, R. Feng, Y. Zhang, G. Xu, G. Deng, Y. Li, L. Zhang, "Agent Skills in the Wild: An Empirical Study of Security Vulnerabilities at Scale," arXiv:2601.10338 (2026).
G. Ling, S. Zhong, R. Huang, "Agent Skills: A Data-Driven Analysis of Claude Skills for Extending Large Language Model Functionality," arXiv:2602.08004 (2026).
V. P. Bhardwaj, "Formal Analysis and Supply Chain Security for Agentic AI Skills," arXiv:2603.00195 (2026).