Usefulness Doesn't Predict Safety: 7.5% of Top Skills Fail the Scan

Notes on SkillAudit: From Fixed-Suite Benchmarking to Skill-Centered Assessment (arXiv:2606.22613) — Dexu Yu, Youhua Li, Zhaoyang Guan, Xianhao Lin, Jining Luan, Zihao Rao, Xuan Lan, Yang Ran, Bo Lan, Naixin Zhai, Hanwen Du, Junchen Fu, and 3 more · June 2026

Note published July 25, 2026 · written by SkillFed’s research pipeline from the paper above · how these notes are made

AI-assisted notes · reviewed by SkillFed Agentic benchmarks Bridge: benchmarks × security

SkillAudit swaps fixed-benchmark grading for skill-centered assessment: feed it an arbitrary skill package, it parses the SKILL.md, and generates capability-aligned tasks that probe exactly what the skill claims to do. Each task runs twice in an isolated sandbox — once with the skill installed, once without — same backbone model, same configuration, same rubric. That pairing is the baseline comparison principle: it isolates what the skill actually contributes from what the backbone could already do alone. A separate two-stage safety pipeline runs in parallel. An LLM first scans the package against 21 predefined risk patterns across five categories; every flagged risk then becomes a live runtime probe — an agent actually attempting the risky action — and only a confirmed trajectory, file change, or network call counts against the skill.

Across 226 top-ranked skills, 23 occupational categories, and six agent/model backbone pairings, the with/without comparison produces a mean pass-rate gain (PRG) of 0.183 in the paper's primary configuration (Codex on GPT-5.4) — pass rate goes from 0.763 without the skill to 0.946 with it. The gain shrinks as the backbone gets stronger: mean PRG runs from 0.248 on the weakest no-skill baseline (Codex/GPT-5.1) down to 0.130 on the strongest (OpenCode/Sonnet 4.6), a strong negative correlation (r=-0.90) between how much a skill helps and how capable the model already was. Efficiency doesn't follow the same curve. Overall efficiency-cost gain is negative (-0.186), and only 2 of 23 occupational categories show a net efficiency win — skills mostly buy pass rate by spending more time and tokens, not less. On safety, the two-stage scan flags 17 of the 226 skills (7.5%) as risky, and utility gain barely correlates with safety score at all (r=0.111): how much a skill helps and whether it's safe turn out to be nearly independent questions.

Key numbers

Skills audited	226 skills, 23 occupational categories
Risky skills flagged	17 of 226 (7.5%)
Mean pass-rate gain (primary config)	0.183 (0.763 → 0.946 pass rate)
Efficiency-cost gain	-0.186 overall; positive in only 2 of 23 categories
Utility–safety correlation	r = 0.111 (near zero)

What this means for skill authors

This is the strongest case yet for evaluating the artifact rather than the leaderboard entry. A fixed benchmark score conflates skill quality with backbone strength, and it can reward a skill just because its tasks happen to resemble the benchmark. A paired with/without run on a fixed backbone is the unit that actually answers the question that matters: does installing this skill help. Any catalog serious about ranking skills should be doing something structurally similar — score the artifact against itself, with and without, instead of scoring the model against a generic suite.

The safety number is the one to sit with. That 7.5% comes from a pool already filtered by popularity — top-ranked packages, not obscure long-tail skills nobody installs. And because utility gain and safety score are nearly uncorrelated, a skill's measured usefulness tells you nothing about whether it would pass a safety scan. A skill that makes an agent dramatically better at a job can still be one scan away from a risky verdict.

It's also the qualitative case for pairing static and dynamic checks instead of trusting either alone. A semantic scan flags patterns that may never fire in practice; a runtime probe by itself would miss risk in code paths a task never touches. SkillFed's dual-tool scanning rests on the same logic — a vetted catalog only means something once 'flagged by a scan' and 'confirmed at runtime' both have to be true before a skill counts as clean.

Skills related to this research

Rtk Rtk helps you set up and troubleshoot Rust Token Killer, a CLI tool that rewrites shell commands into token-efficient versions for AI agents. Install via Homebrew, script, or Cargo, then initialize RTK for your specific agent—Claude Code, Codex, Gemini CLI, Cursor, Copilot, Windsurf, Cline, or OpenCode. Use direct `rtk` commands like `rtk git status` or `rtk read` when automatic hooks aren't available.★ 38 Ios Networking This skill teaches you to build robust network communication in iOS and macOS apps using URLSession with async/await and structured concurrency. You'll learn practical patterns for HTTP requests, REST API clients, file transfers, streaming, and handling network errors—plus architecture strategies like middleware, API client protocols, and testable designs.★ 932

Related notes

References

Yu, D. et al. SkillAudit: From Fixed-Suite Benchmarking to Skill-Centered Assessment. arXiv:2606.22613 (2026).
Harbor — the sandbox environment SkillAudit extends for isolated skill execution.