Coding agents violate their own skill's embedded logic in up to 70% of test cases

Notes on SLBench: Evaluating How LLM Agents Follow Logical Relations in Skills (arXiv:2607.09016) — Xuan Chen, Chengpeng Wang, Lu Yan, Xiangyu Zhang · July 2026

Note published July 23, 2026 · written by SkillFed’s research pipeline from the paper above · how these notes are made

AI-assisted notes · reviewed by SkillFed Agentic benchmarks

SkillLogic is a static-analysis framework that reads an agent skill file and extracts the logical relations binding its instructions together: preconditions that gate an action, postconditions that stay unmet until a follow-up step runs, constraints that narrow how an allowed action can be carried out, fallbacks that only kick in after a primary route fails, plus conjunctions, exceptions, overrides, and conflicts — eight relation types in all. Run against 5,224 public skills pulled from SkillsMP, the scanner found that 70% carry at least one such relation — most skills already demand that an agent reason about sequencing and precedence, not just execute a single command. The highest-confidence, highest-impact, locally-gradable relations feed into SLBench: 86 audited cases, each built around a seeded repository and a grader that checks execution artifacts — files written, configs changed, logs left behind — rather than the agent's own account of what happened.

Running Codex and Claude Code across six backbone models on SLBench, unsafe outcomes ranged from 35% to 70.2% of cases. Codex on GPT-5.5 was worst, with 70.2% unsafe; Claude Code on Sonnet 4.6 posted the highest safe rate of any configuration (44.2%) and still landed at 43.0% unsafe, while Opus 4.7's lower unsafe rate (35.1%) came partly from more runs being graded inconclusive rather than safe. Precondition, conflict, and override relations broke down most often — agents acted before a required gate cleared, or followed a weaker permission over a stronger blocker — while fallback violations were comparatively rare. A 12-case human audit found zero instances where careful human readers judged the underlying skill text ambiguous, yet rewriting those same skills for clarity only cut violations from 11 of 12 to 5 of 12, evidence that both agent capability and low-salience wording share the blame. SLGuard, a lightweight scaffold that forces the agent to build and check off a relation checklist before declaring the task done, converted 7 of 11 targeted violations into safe outcomes — a 63% reduction — though precondition and exception failures proved harder to close with prompting alone.

Key numbers

Skills with ≥1 logical relation (of 5,224 scanned)	70%
Unsafe rate, worst config (Codex + GPT-5.5)	70.2%
Unsafe rate, best safe-rate config (Claude Code + Sonnet 4.6)	43.0%
Violations fixed by SLGuard checklist (of 11 targeted)	63% (7/11)
Audit cases rated human-ambiguous (of 12)	0

What this means for skill authors

The prevalence number is the one skill authors should sit with: if 70% of skills already encode gating, scoping, or precedence logic, most authors are — whether they intend it or not — asking an agent to enforce a small state machine, not just follow a checklist. The fix implied by the human-audit result isn't to write more text; it's to make gates and completion conditions structurally salient rather than folding them into prose. A cleanup requirement stated as its own numbered step survives better than one buried mid-paragraph under a different heading.

For anyone selecting or vetting skills, the finding that matters is the disconnect between how a skill reads and how an agent executes it. The same skill text that a human resolves correctly on first read can still lead an agent to leave sensitive files on disk or greenlight a blocked action, because the failure shows up only in execution artifacts, never in the transcript the agent hands back. That's a structural argument for evaluating skills by what agents actually do when they run them rather than by how well the description reads — checking real execution behavior alongside the written instructions, which is the gap dual-tool scanning in a vetted catalog is meant to close.

SLGuard's result is more informative for what it fails to fix than for what it fixes. A relation checklist clears out constraint, override, and postcondition failures — cases where the safe behavior is a concrete scope boundary or a completion step you either did or didn't do. It does far less for preconditions and exceptions, which require tracking state and reasoning about a narrow carve-out rather than ticking a box. That split is a useful constraint for anyone building lightweight guardrails: prompting closes the easy half of the gap; the gating half likely needs runtime state-checking, not more instructions.

Skills related to this research

context7-auto-research This skill detects when you're asking about libraries, frameworks, APIs, or code examples and automatically retrieves up-to-date documentation from Context7 instead of relying on potentially outdated training data. It identifies the relevant library, searches Context7, and integrates current documentation into your response with practical code examples.★ 16 drizzle-orm Drizzle ORM is a lightweight, type-safe TypeScript ORM that treats SQL as a first-class citizen. This skill covers schema definition, database connections across PostgreSQL, SQLite, and Turso, relational queries, and performance patterns like proper indexing and pagination.★ 202 Qt Ui Design Qt UI Design guides you through building interfaces for Qt/QML projects across desktop, web, and embedded platforms. It walks you through target platform, screen geometry, design system selection, and content hierarchy before you start, then applies established UX principles—layout, typography, motion, and interaction patterns—to keep your design coherent and usable.★ 322 drizzle Drizzle is a TypeScript-first ORM built for compile-time type safety and edge-runtime performance. Define schemas with SQL-like syntax, execute queries with full type inference, and manage one-to-many and many-to-many relations without runtime dependencies.★ 62

Related notes

References

Chen, Wang, Yan & Zhang (2026). SLBench: Evaluating How LLM Agents Follow Logical Relations in Skills. arXiv:2607.09016.
Debenedetti et al. (2024). AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents. NeurIPS Datasets and Benchmarks Track.
Wallace et al. (2024). The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions. arXiv preprint.
Jiang et al. (2026). SoK: Agentic Skills — Beyond Tool Use in LLM Agents. arXiv:2602.20867.