Checking the repo, not just the SKILL.md, cuts flagged 'malicious' skills from 46.8% to 0.52%

Notes on Malicious Or Not: Adding Repository Context to Agent Skill Classification (arXiv:2603.16572) — Florian Holzbauer, David Schmidt, G. Gegenhuber, S. Schrittwieser, Johanna Ullrich · March 2026

Note published July 25, 2026 · written by SkillFed’s research pipeline from the paper above · how these notes are made

Researchers crawled 238,180 unique agent skills from ClawHub, Skills.sh, SkillDirectory, and GitHub — the largest cross-platform census of the skill ecosystem so far. Every skill passed through three lenses: each marketplace's own scanner, an independent Cisco skill scanner, and the authors' own LLM-based classifier. For skills flagged high-risk by both an external scanner and their model, the team went a step further and checked the skill against the GitHub repository it actually ships from — README, code, commit history — rather than judging the SKILL.md file in isolation.

Marketplace-reported malicious rates swing from 46.8% on ClawHub down to 6.0% on SkillDirectory, and the five scanners tested barely agree with each other: among the 27,111 skills all five tools analyzed in common, only 33 (0.12%) got flagged by every one of them. Aggregating flags at the repository level roughly doubles the apparent risk, too — 19.3% of individual skills get flagged versus 45.9% of the repositories hosting them, since one bad skill is enough to tag an entire repo as malicious. Once flagged skills are checked against their home repository, the suspicious set nearly disappears: of 2,887 scanner-flagged skill-repository pairs fully evaluated, only 15 (0.52%) still looked malicious once repository alignment and maturity were factored in, and manual reviewers who checked a sample of flagged repos found nearly all of them benign. The same repository-level lens surfaced real risk elsewhere — 121 skills route through seven abandoned, hijackable GitHub repositories, and a dozen live API credentials, including NVIDIA, ElevenLabs, and MongoDB tokens, sit exposed inside published skills.