Metadata Alone Wins 86% of Skill-Discovery Matchups

Notes on Under the Hood of SKILL.md: Semantic Supply-chain Attacks on AI Agent Skill Registry (arXiv:2605.11418) — Shoumik Saha, Kazem Faghih, S. Feizi · May 2026

Note published July 25, 2026 · written by SkillFed’s research pipeline from the paper above · how these notes are made

SKILL.md files — the natural-language front matter that tells an agent when and how to use a packaged capability — sit upstream of three separate decisions: whether a skill gets found, whether it beats a rival for selection, and whether it clears a security check. Researchers tested all three against real skills pulled from ClawHub, a live agent-skill registry, drawing on 100 skills across five categories (email, travel, tax, health, prompt). Discovery got the beam-search-and-gradient treatment: short adversarial trigger phrases, appended to the SKILL.md text, tested whether embedding-based retrieval could be steered toward a skill without touching a line of its code. Selection worked differently — functionally identical skill pairs, differing only in how the description was framed (exaggerated capability claims, directive language, recency signals, trust claims), put in front of four LLM agents asked to pick one. Governance was the toughest test: a three-stage vetting pipeline — a static scanner, an LLM reviewer, a VirusTotal malware pass — run against admittedly malicious skills rewritten with wording-only evasion tactics like paraphrasing and context-window overflow.

The trigger attack is cheap and effective. Adversarial phrases roughly 1% the length of an average SKILL.md file pushed a skill's win rate against OpenAI's embedding model to 86% in pairwise matchups and landed it in the Top-10 results 80% of the time; triggers optimized against one embedding model still transferred to another at 79-80%. Selection needed no exploit at all — framing alone, with no functional edge attached, got adversarial variants picked in 77.6% of paired trials on average across the four LLM agents tested, and every model chose the adversarial option above chance. Governance evasion was uneven but sometimes total: paraphrasing an already-flagged malicious skill dodged a blocking verdict roughly 74% of the time, and padding a SKILL.md file with enough filler to trigger context-window overflow got 87% of malicious variants labeled clean outright, with none flagged as malicious at all.