More Than Half of 13,728 Real-World Skills Hide a Critical Risk in Plain Prose

Notes on Semia: Auditing Agent Skills via Constraint-Guided Representation Synthesis (arXiv:2605.00314) — Hongbo Wen, Ying Li, Hanzhi Liu, Chaofan Shou, Yanju Chen, Yuan Tian, Yu Feng · May 2026

Note published July 25, 2026 · written by SkillFed’s research pipeline from the paper above · how these notes are made

Every agent skill is two artifacts stapled together: a structured half that declares callable actions, and a prose half that tells the agent when to fire them — prose that gets reinterpreted fresh, probabilistically, on every invocation. Static analyzers parse the structured half and ignore the prose; LLM-based auditors read the prose but can't reproducibly prove a tainted input reaches a dangerous sink. Semia lifts each skill into the Skill Description Language (SDL) — a Datalog fact base capturing triggers, data flow, authorization gates, and documented claims. It generates that fact base through Constraint-Guided Representation Synthesis (CGRS), a propose-validate-verbalize loop that keeps refining a candidate until it's both structurally sound and faithful to the source text. Once the facts exist, checks for indirect prompt injection, secret leakage, or a confused deputy become plain Datalog reachability queries.

Run against 13,728 real-world skills pulled from public marketplaces, Semia renders every one of them auditable and finds that more than half carry at least one critical semantic risk. On a 541-skill benchmark hand-labeled by two annotators (Cohen's kappa 0.83; 301 confirmed risky, 240 clean), Semia reaches 97.7% recall and 84.5% precision, an F1 of 90.6% — against 13.6% recall for VirusTotal-style signature scanning and 52.6% recall for the marketplace's own heuristic-plus-LLM scanner, ClawScan. Removing the SDL representation and falling back to a single LLM pass costs 19.9 F1 points; turning off the iterative refinement loop alone costs another 4.0, showing the structured intermediate representation is doing real work rather than dressing up an LLM's opinion. The same audit surfaced 17 exploitable zero-day vulnerabilities, confirmed and responsibly disclosed through the marketplace's maintainers.