90% of a Skill Marketplace's Most-Downloaded Skills Fail a Rigorous Security Audit

Notes on SkillProbe: Security Auditing for Emerging Agent Skill Marketplaces via Multi-Agent Collaboration (arXiv:2603.21019) — Zihan Guo, Zhiyu Chen, Xiaohang Nie, Jianghao Lin, Yuanjian Zhou, Weinan Zhang · March 2026

Note published July 23, 2026 · written by SkillFed’s research pipeline from the paper above · how these notes are made

SkillProbe is a three-stage, multi-agent auditing pipeline built for agent skill marketplaces like ClawHub. First, a Gatekeeper stage filters for compliance violations and known-malicious patterns. Then an Alignment Detector checks semantic-behavioral alignment — whether a skill's declared behavior actually matches what its code does — sorting each skill into a four-way match / over-declaration / under-declaration / mixed classification. The final stage, a Flow Simulator, models combinatorial risk: individually benign skills chaining into dangerous behavior once an agent invokes several together. Brute-forcing every possible combination would mean an O(2^N) search, so the simulator instead matches against a fixed rule set, keeping cost linear in catalog size — O(N × number of rules). The whole pipeline is itself packaged as skills that drive specialized auditing agents, a design the authors call "Skills-for-Skills." The team ran it across 2,500 real skills pulled from ClawHub, then separately re-audited the top 20 most-downloaded skills with 8 different LLM series to see how verdicts shift depending on which model is doing the judging.

The results argue against treating popularity as a safety signal. Across the full 2,500-skill corpus, only 9.9% (247 skills) came back fully clean, 59.8% passed the baseline audit but carried minor flaws, and 30.4% (759 skills) were flagged Conditional over real security violations. Among the most-downloaded skills specifically, over 90% failed rigorous auditing outright. The composability check turned up something sharper: the 499 skills flagged high-risk didn't sit in separate silos — they formed one giant connected component, linked by 75,373 risk edges: 32,328 (43%) data-exfiltration paths, 28,441 (38%) fact-poisoning paths, 8,637 command-injection paths. The choice of auditing model moved the outcome substantially: Sonnet 4.6 was the strictest judge, approving only 55% of the top-20 skills and flagging 45% Conditional, while Nex-N1.1 and GPT-5.2-Codex approved up to 95% of the same set, with per-audit runtimes ranging from 18.5 seconds (Gemini Flash) to 282 seconds (Nex-N1.1).