Malicious-Skill Detection Hits 0.920 F1 at $0.006 per Skill

Notes on SkillSieve: A Hierarchical Triage Framework for Detecting Malicious AI Agent Skills (arXiv:2604.06550) — Yinghan Hou, Zongyou Yang · April 2026

Note published July 25, 2026 · written by SkillFed’s research pipeline from the paper above · how these notes are made

A regex scanner can read a skill's code but not the English prose in its SKILL.md — exactly where prompt injection and social-engineering instructions like to hide. Formal static analyzers read code more rigorously and still can't parse prose. SkillSieve's answer isn't a smarter single scanner; it's three escalating layers, each one filtering for the next. Layer 1 is a zero-cost static analysis pass — regex, AST parsing, metadata heuristics — that clears roughly 86% of skills in under 40ms apiece. What's left goes to Layer 2, an LLM that splits judgment into four parallel sub-checks (intent alignment, permission justification, covert-behavior detection, cross-file consistency) rather than one monolithic prompt. Skills that still look risky hit Layer 3: a multi-model jury — GLM-5.1, Qwen3-235B, DeepSeek-V3.1 — that votes independently and only debates when the votes split. The full pipeline ran end to end on a single $440 ARM single-board computer against 49,592 real skills pulled from OpenClaw's ClawHub registry (a 2026-04-04 snapshot spanning 16,797 authors), plus adversarial samples built around five evasion techniques.

On a 390-skill hand-labeled benchmark, the full pipeline lands F1 0.920 — precision 0.912, recall 0.929 — at an average $0.006 per skill and a 0.015 false-positive rate in deployment. Each layer earns its place in the chain. Layer 1 alone catches everything (recall 1.00) but floods the queue with noise (precision 0.348, F1 0.516). Layer 2's decomposition lifts precision to 0.675 by clearing 78 of Layer 1's 105 false positives. The jury pushes precision to 0.912, overturning 24 more false positives and kicking the 4 cases where jurors split 2-1 to human review. All five adversarial evasion techniques tested — homoglyph substitution and time-delayed payloads among them — got caught. An optional XGBoost pre-filter trims downstream LLM calls by 32% for a 1.6-point F1 tradeoff, recall unmoved. Ported to Feishu/Lark with a lightweight adapter, the same pipeline scanned 52 enterprise packages: Layer 2 corrected 13 of Layer 1's 14 false-positive flags — mostly API timestamps and documentation strings misread as obfuscated payloads — and still caught one genuine plaintext-credential leak.