SKILL.md alone catches 0% of malicious skills; the full package recovers 67%

Notes on Structured Security Auditing and Robustness Enhancement for Untrusted Agent Skills (arXiv:2604.25109) — Lijia Lv, Xuehai Tang, Jie Wen, Jizhong Han, Songlin Hu · April 2026

Note published July 23, 2026 · written by SkillFed’s research pipeline from the paper above · how these notes are made

Agent Skills bundle a SKILL.md manifest with scripts, reference documents, and repository context, so auditing one before it loads means reviewing several files as a unit rather than screening a single prompt. Sorting a package into benign, suspicious, or malicious takes four stages: pull role-weighted evidence out of every file, route only the uncertain cases to a semantic verifier, arbitrate between competing attack-chain readings — a hidden override, a disguised data transfer, a remote bootstrap dependency — then check the verdict against the package's semantics-preserving rewrites, escalating to "malicious" only once every rewrite agrees. That pipeline is SkillGuard-Robust, tested on SkillGuardBench plus two public-ecosystem extensions, across five evaluation views spanning 254 to 404 packages each.

Two results stand out. Cut the input down to SKILL.md alone on the alpha set, holding everything else fixed, and the authors' own pipeline's malicious recall drops to zero. Hand back the full package and recall on that same set climbs to roughly two-thirds, 67%; the manifest by itself carries almost none of the signal, and full-package context is what makes detection possible at all. The second result is a more specific failure mode: given full context, a strong baseline correctly flags risky packages as non-benign but routinely resolves them as merely "suspicious" rather than "malicious," hitting 100% flagged-accuracy alongside 0% malicious recall on one held-out batch. SkillGuard-Robust's staged verification and rewrite-consistency checks close that gap, reaching 97.30% overall exact match and 98.33% malicious recall on the 404-package held-out aggregate, and 99.66% exact match with 100% malicious recall and 100% attack-consistency on the 254-package external-ecosystem view built from repositories outside the original benchmark family.