Best Skill Scanner Catches 17% of Cross-Modal Attacks — a Distilled 7B Model Hits 88%

Notes on SkillMutator: Benchmarking and Defending Language-and-Code Cross-modal Attacks on LLM Agent Skills (arXiv:2606.14154) — Youngduk Kim, Minkyoo Song, S. Shin · June 2026

Note published July 24, 2026 · written by SkillFed’s research pipeline from the paper above · how these notes are made

SkillMutator targets a blind spot in Agent Skills security: attacks that only emerge from how a skill's natural-language SKILL.md and its executable code interact, not from either file read on its own. A SKILL.md can read as a completely benign workflow while an implicit directive buried in the prose steers the agent to exfiltrate sensitive files — the accompanying script stays clean when checked by itself. The benchmark turns this into a moving target: an adversarial mutation process iteratively rewrites malicious skills against scanner feedback until the injected behavior reads as indistinguishable from a legitimate workflow, covering 13 attack categories across four threat groups that range from technical exploits like exfiltration and privilege escalation to semantic tricks like brand hijacking and supply-chain moves like configuration weakening. The training corpus runs 1,219 mutations generated over 68 community-authored skills; evaluation draws 48, 63, and 76 surviving mutations from three adversarial oracles of increasing strength (GPT-4o-mini, GPT-5.4-mini, and GPT-5.4) applied against 17 of Anthropic's official Agent Skills.

Existing detection barely registers against this benchmark: an open-source scanner (skill-security-scan) catches 2.1%-7.9% of mutations, a commercial LLM-based scanner (Snyk Agent Scan) reaches 9.2%-16.7%, and a rule-based commercial product (SkillScan API) manages 0%-1.3%. The fix is a four-phase reasoning-trajectory distillation pipeline — ground the skill's stated purpose, flag out-of-scope actions, reason against nine security principles, then label the attack category — that transfers a frontier model's reasoning traces into a small open-weight model, Qwen2.5-Coder-7B-Instruct. Zero-shot, that 7B model catches only 17.1% of mutations on the hardest subset (n=76); after distillation it hits 88.2%, ahead of GPT-4o-mini's 23.7% and GPT-5.4-mini's 79.0%, and within two points of frontier GPT-5.4's 86.8%.