80% of agent skills in a 49,943-skill registry don't do what they claim — most of it is sloppiness, not malice

Notes on Behavioral Integrity Verification for AI Agent Skills (arXiv:2605.11770) — Yuhao Wu, Tung-Ling Li, Hongliang Liu · May 2026

Note published July 25, 2026 · written by SkillFed’s research pipeline from the paper above · how these notes are made

AI-assisted notes · reviewed by SkillFed Skill security Bridge: security × benchmarks

Most skill safety work checks prompts and runtime actions, never the skill artifact itself against its own description. Behavioral integrity verification (BIV) names that missing check: a typed comparison between what a skill declares and what it actually does, run over a shared capability taxonomy of 29 capability types spanning code, natural-language instructions, and metadata. The declared side comes from parsing manifests plus LLM-assisted extraction, constrained by a hallucination-control filter that requires a quoted source span for every claimed capability. The actual side comes from deterministic analysis — inter-procedural taint analysis for Python, regex extraction for JavaScript and shell — plus LLM parsing of natural-language instructions. Both sides land in the same typed taxonomy, producing one structured evidence record per skill that feeds three downstream jobs: a deviation taxonomy, a root-cause classifier, and a malicious-skill detector.

Run across 49,943 skills pulled from the OpenClaw registry, the deviation taxonomy turns up a wide description-implementation gap: 80.0% of skills deviate from what they declare, roughly split between undeclared capabilities quietly present in the code and declared capabilities that were never implemented. Root-cause classification then sorts those deviations by intent: 81.1% trace to ordinary developer oversight, 18.9% look adversarial, and 5.0% of all skills carry a predicted multi-stage attack chain. Clustering surfaces four compound-threat patterns worth naming individually: exfiltration chains, remote-code-execution chains, code obfuscation, and data-lineage violations. How adversarial each one is varies sharply by category — it's not one registry-wide rate. Feeding this structured evidence into an LLM judge, with an explicit override rule for cases where compound-threat flags coincide with undeclared high-risk capabilities, gets malicious-skill detection to an F1 of 0.946 on a 906-skill benchmark — ahead of both the rule-based and single-pass LLM baselines it's tested against.

Key numbers

skills audited from the OpenClaw registry	49,943
of skills deviate from their declared behavior	80.0%
of deviations trace to adversarial intent (vs. 81.1% oversight)	18.9%
of skills carry a predicted multi-stage attack chain	5.0%
malicious-skill detection F1, 906-skill benchmark	0.946

What this means for skill authors

The headline number reads alarming until the root-cause split lands: four in five deviations are oversight, not malice. That's the more useful fact for anyone deciding what to do about it. A skill that quietly touches the filesystem it never mentioned is usually a stale manifest, not a trojan. Treat every gap as a security incident and reviewers drown — and learn to ignore the real ones. Separating "needs a documentation fix" from "needs a security review" is what makes auditing at registry scale tractable at all.

The category-level detail is the part worth sitting with longer than the topline rate. Adversarial fraction swings from roughly 1 in 10 deviations for filesystem-related gaps up to 9 or 10 in 10 for instruction-level threats and code obfuscation. A single risk score per skill would flatten all of that into noise. The compound-threat categories matter because they're combinations, not single flags: a skill that both reads credentials and phones out is a different animal than one that does either alone. That distinction only shows up if declared and actual capabilities get compared as structured, typed sets — not eyeballed.

The general shape here — verify the artifact against its own claims, not just its runtime behavior — is the same logic behind scanning a skill through more than one independent method before it reaches a catalog. A manifest is a claim the author makes about their own code; nothing in it is self-verifying, and these numbers show how often that gap opens, at real registry scale.

Skills related to this research

playwright-scanner playwright-scanner automates accessibility testing by running behavioral checks against live web pages through Playwright. It identifies WCAG compliance gaps and accessibility barriers, helping teams catch usability issues before users encounter them. Powered by Gemini for intelligent analysis.★ 370 architecture-decision This skill guides you through authoring Architecture Decision Records (ADRs) that capture major technical choices with full context. It validates decisions against existing architectural stances, engine compatibility, and GDD requirements, then generates a structured record with status, dependencies, and consequences tracked. Use it to retrofit missing sections into existing ADRs or create new ones from scratch.★ 23,424 ml-mlops ML MLOps guides you through building auditable, repeatable machine learning workflows. It covers experiment tracking, model versioning and governance, pipeline orchestration across Kubernetes, AWS, GCP, and Azure, CI/CD automation, and production monitoring to catch drift and quality issues.★ 49 knowledge-base Create a compounding knowledge base—a workspace wiki of plain markdown pages linked together with wikilinks, designed to grow denser and more interconnected as you add sources. Ingest articles, documents, and notes into versioned raw files, then synthesize them into organized pages with frontmatter, cross-references, and provenance tracking. Run automated linting to catch broken links, orphaned pages, and schema drift, keeping your knowledge graph clean and queryable.★ 1,423

Related notes

References

Wu, Y., Li, T.-L., & Liu, H. (2026). Behavioral Integrity Verification for AI Agent Skills. arXiv:2605.11770.