534 of 3,984 public skills carried a critical security flaw — SkillTester grades against that baseline

Notes on SkillTester: Benchmarking Utility and Security of Agent Skills (arXiv:2603.28815) — Leye Wang, Zixing Wang, Anjie Xu · March 2026

Note published July 25, 2026 · written by SkillFed’s research pipeline from the paper above · how these notes are made

SkillTester is a comparative test harness for agent skills — the packaged, SKILL.md-based capabilities now shipped across Claude Code, OpenAI Codex, GitHub Copilot, and OpenClaw. Its core rule, what the paper calls the comparative utility principle, is simple to state and strict to enforce: every task runs twice, once with the skill disabled and once enabled, on the same model and environment, and a skill only earns credit if it's actually invoked. That gate stops a model's own baseline competence from getting credited to the skill. Utility tasks split into common functional cases — the skill's stated use — and edge cases covering failure handling; each has to resolve in both conditions with a documented pass criterion before it's admitted to the benchmark. Security gets scored separately, through a controlled security probe suite rather than paired execution, organized into three groups — abnormal behavior control, permission boundary, sensitive data protection — built on one premise: a skill's SKILL.md claims and safety badges are unverified assertions until tested against actual code and behavior.

Scoring collapses to three numbers. A task gets 0 if the skill isn't invoked or fails, 100 if the skill succeeds where the baseline fails — a clean win — and if both succeed, the score is set by relative token and time cost: log-scaled around a neutral point of 50, with a floor of 20 so a successful-but-slower run never gets crushed toward zero. Security groups are each scored as a plain pass-rate percentage, averaged unweighted into one number, then bucketed into a status label — Pass requires a perfect 100, Caution covers 80 up to 100, Risky is anything under 80. None of this exists in a vacuum: the paper cites a February 2026 Snyk audit of 3,984 public skills pulled from ClawHub and skills.sh that found 534 carrying at least one critical security issue, 1,467 with a flaw of any severity, and 76 confirmed-malicious payloads. Popularity and self-reported badges, in other words, aren't a substitute for behavioral testing.