The best skill scanner hits 98% recall — and still flags 937 of 4,000 safe skills as malicious

Notes on MalSkillBench: A Runtime-Verified Benchmark of Malicious Agent Skills (arXiv:2606.07131) — Wenbo Guo, Wei Zeng, Chengwei Liu, Xiaojun Jia, Yijia Xu, Lei Tang, Yong Fang, Yang Liu · June 2026

Note published July 24, 2026 · written by SkillFed’s research pipeline from the paper above · how these notes are made

AI-assisted notes · reviewed by SkillFed Skill security Bridge: security × benchmarks

MalSkillBench builds ground truth for detecting malicious agent skills — the SKILL.md packages that bundle natural-language instructions, scripts, and tool permissions for coding agents like Claude Code and Gemini CLI — by actually running them rather than trusting a label. A closed-loop Generate-Verify-Feedback pipeline seeds a three-dimensional taxonomy (attack vector × behavior × insertion strategy, 108 cells) with real malicious PyPI code and prompt-injection corpora, then drops each candidate skill into a Docker sandbox with syscall monitoring and an LLM judge; only skills whose malicious behavior actually fires get admitted. The result is 3,214 verified generated skills, plus 703 confirmed skills pulled from the wild and 27 drawn from existing tool test suites — 3,944 malicious skills in total, paired with 4,000 benign skills pulled from ClawHub for false-positive testing.

The realizability numbers alone tell a story: code injection fires reliably (94.5% verification yield) while prompt injection is far shakier (75.8%, dropping to 62.5% when steganographically hidden) — the same instruction-layer fragility that later makes it hard to catch. The 703 wild skills collected turn out to be almost entirely one story: 86.6% deliver Malware Delivery via a fake-prerequisite dependency, and 81% trace to just two accounts running a single cryptocurrency-theft campaign; a small 1.7% tail instead attacks the agent's control plane (session hooks, identity, instruction hierarchy) rather than the host. Against the full benchmark, the strongest detector (Sentry Skill Scanner, full mode) reaches 98.4% recall and 88.6% F1 but still misflags 937 of 4,000 benign skills. Score everything on wild data alone, though, and the rankings flip: VirusTotal's recall jumps 66 points, from 21.6% to 87.9%, once you switch to the skewed subset.

Key numbers

Benchmark scale	3,944 malicious skills across 108 taxonomy cells
Code vs. prompt-injection realizability	94.5% vs 75.8% verification yield
Wild sample concentration	86.6% one behavior, 81% from two accounts
Best detector's false positives	937 of 4,000 benign skills flagged (at 98.4% recall)
Wild-only evaluation bias	recall swings up to 66 points (VirusTotal: 21.6% to 87.9%)

What this means for skill authors

The paper's central diagnosis is that a skill's maliciousness is relational, not localized: whether a requested dependency, shell command, or redefined role is justified by the task the skill advertises. That's exactly why supply-chain scanners reading code alone and prompt-injection filters reading text alone each see only half the artifact, and why naive OR/AND combinations of the two don't recover the missing relationship (OR inherits the false positives, AND collapses recall below 6%). It's a direct argument for scanning code and instructions jointly rather than bolting a code scanner onto a prompt filter — the qualitative case for dual-tool review over single-domain checks. SkillFed doesn't have a MalSkillBench-scale number to cite here, but the mechanism the paper identifies is the one any vetted catalog has to defend against.

For skill authors, the practical read is narrower than the headline stats suggest: nearly nine in ten wild attacks are still a single pattern of dependency impersonation — a benign-looking skill that asks the agent to install a "prerequisite" dependency first, and the real payload lives there. Anyone building or reviewing a skill should treat an install-before-you-run step as the single highest-scrutiny line in a SKILL.md. The rarer but more interesting threat is the tail that manipulates the agent itself, not the host: a SessionStart hook that persists across every future session, or a helper script that rewrites the agent's own system prompt and plants an override above the skill's stated task to seize the instruction hierarchy. Neither has a real equivalent in npm or PyPI malware, and both are the hardest category to generate and detect — worth watching as skill ecosystems grow past their current, impersonation-dominated phase.

There's a methodology lesson too, one that applies past this specific paper: any claim about a scanner's real-world accuracy that's measured only against skills caught in the wild is measuring against whatever one or two campaigns happened to dominate that sample, not the attack surface as a whole. A detector, or a vetting pipeline, that looks strong on wild-only numbers can still be blind to the instruction-layer and agent-control attacks that haven't shown up in bulk yet, precisely because they're harder to pull off.

Skills related to this research

codspeed-optimize This skill acts as an autonomous performance engineer, using CodSpeed benchmarks and flamegraph analysis to iteratively optimize code. It measures baseline performance, identifies bottlenecks through flame graphs, applies targeted changes, and validates improvements across simulation and walltime modes. The skill handles the full optimization loop—from establishing baselines to comparing runs and reporting gains—stopping only when significant improvements plateau or the user decides to halt.★ 235 codspeed-setup-harness This skill guides you through configuring benchmarks and integrating CodSpeed into your project. It covers language-specific harnesses (divan, criterion, pytest-benchmark, vitest, go test, Google Benchmark) and a universal exec mode for any language, helping you identify performance-critical code and establish repeatable performance tracking.★ 235 performance-attribution Break down why your portfolio outperformed or underperformed its benchmark by isolating contributions from sector positioning, individual stock picks, and market timing. Performance Attribution uses Brinson decomposition, factor regression, and timing models to reveal the true drivers of excess return across multiple time periods.★ 28,096 performance-attribution Decompose active returns into the sources that drove outperformance or underperformance—allocation decisions, security selection, and their combined effects. Supports equity attribution via Brinson-Fachler, factor-based decomposition, fixed-income yield and curve analysis, and currency impact for international portfolios.★ 159

Related notes

References

Guo, W. et al. MalSkillBench: A Runtime-Verified Benchmark of Malicious Agent Skills. arXiv:2606.07131 (2026).
Liu et al. Wild-collected malicious agent skill dataset (157 samples), cited as the prior public academic benchmark, 2026.
Guo et al. IntelliGraph: malicious PyPI package dataset with source code, call graphs, and attack-chain annotations (3,026 packages), 2024.
Jiang et al. WildJailbreak: in-the-wild jailbreak prompt corpus, 2024.
Andriushchenko, M. et al. AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents. ICLR 2025.