Agent-skill security scanners agree on just 0.69% of all skills — each is watching a different layer

Notes on ClawHub Security Signals: When VirusTotal, Static Analysis, and SkillSpector Disagree (arXiv:2606.01494) — Vincent Koc, Patrick Erichsen, J. Tomlinson, A. Rivera, Michael Appel, N. Paz · May 2026

Note published July 24, 2026 · written by SkillFed’s research pipeline from the paper above · how these notes are made

AI-assisted notes · reviewed by SkillFed Skill security

ClawHub Security Signals pairs each of 67,453 latest public OpenClaw skill versions — redacted SKILL.md content plus sanitized bundled files where present — with a ClawScan verdict (clean, suspicious, or malicious) and independent evidence from three scanner families: VirusTotal, static heuristic analysis, and NVIDIA's SkillSpector, which raises semantic agentic-risk advisories rather than malware-reputation signals. The release isn't trying to estimate how many skills are actually malicious. It's asking a narrower question: pointed at the same skills, do these three scanners even agree with each other?

They mostly don't. Any two scanners overlap on at most 10.4% of their combined positive flags, and only 0.69% of all skills draw a flag from all three — 81.9% of flagged skills show up on exactly one scanner and nowhere else. That split isn't noise; it tracks attack surface. SkillSpector dominates the suspicious middle tier, raising advisories on 75.3% of suspicious-verdict rows, but catches only 6.8% of confirmed malicious rows — there VirusTotal takes over, at 72.8%. Almost a quarter of malicious verdicts, 24.3%, carry no positive signal from any of the three scanners at all; the registry reached those calls through provenance and moderation context, not scanner detection.

Key numbers

Skill versions in the dataset	67,453
Max pairwise scanner overlap	10.4%
Flagged by all three scanners	0.69%
Flagged by exactly one scanner	81.9%
Malicious verdicts with zero scanner signal	24.3%

What this means for skill authors

For skill authors, the read is blunt: clearing one scanner proves nothing about the others. A skill that passes a reputation check on its bundled files can still draw a SkillSpector advisory for asking the agent to overreach — too much tool authority, or a trigger vague enough to fire on almost anything. A skill with no semantic red flags at all can still ship malicious code that only a reputation-style scanner would catch. Each detector watches one layer of the surface, not the whole skill.

For registries, that's the strongest case yet against single-scanner allow/block gates. A catalog running only static analysis misses most of what semantic, agentic-risk review catches; a catalog running only semantic review misses reputation-flagged malware sitting in the bundled files. That's the qualitative logic behind SkillFed's own dual-tool scanning of its vetted catalog — no single detector spans the risk surface, so catalog trust has to come from combining independent signals, not from picking the one with the best track record.

One caveat the paper is upfront about: these are silver labels, not human-adjudicated ground truth. The ClawScan verdict under study is itself mostly LLM-produced, and SkillSpector is partly LLM-based too — some of the correlation between them could just as easily reflect shared machinery as independent confirmation. Chance-corrected agreement (Cohen's kappa) stayed in the slight range on the Landis-Koch scale (0.045 to 0.082) across every scanner pair. Read the 10.4% overlap figure as proof that no single automated scanner works as a security boundary on its own, not as a validated base rate for how many skills are actually dangerous — the study explicitly isn't measuring prevalence.

Skills related to this research

Statistical Analysis Statistical Analysis guides you through test selection, assumption verification, and effect size reporting for academic research. It covers frequentist methods like t-tests, ANOVA, and regression alongside Bayesian approaches, with specialized workflows for survival analysis, count models, and reliability assessment.★ 284 Ttscn Ttscn converts Chinese and multilingual text to natural speech across 11 cloud backends, from free Edge TTS to premium providers like ElevenLabs and OpenAI. Choose by use case—short video, audiobook, enterprise SSML, or lowest cost—with support for word-level timestamps, pause markers, and pronunciation overrides.★ 24 Daytona Dev Daytona Dev is a development platform designed to accelerate coding tasks and project setup. It provides integrated tools and automation to help developers manage their workflows more efficiently, reducing setup time and enabling faster iteration on code.★ 1,925 Context7 Context7 is an MCP server skill that connects you to library documentation, API references, and setup guides through mcporter in stdio mode. Query libraries like React to find specific documentation, resolve library identifiers, and access up-to-date technical resources without extra configuration.★ 1,538

Related notes

References

Koc, Erichsen, Tomlinson, Rivera, Appel & Paz — ClawHub Security Signals: When VirusTotal, Static Analysis, and SkillSpector Disagree (arXiv:2606.01494, 2026)