Checking what a value is for, not just whether it changed, cuts drift false alarms from 40% to zero

Notes on Skill Drift Is Contract Violation: Proactive Maintenance for LLM Agent Skill Libraries (arXiv:2605.10990) — Linfeng Fan, Yuan Tian, Ziwei Li, Zhiwu Lu · May 2026

Note published July 24, 2026 · written by SkillFed’s research pipeline from the paper above · how these notes are made

AI-assisted notes · reviewed by SkillFed Skill evolution Bridge: evolution × benchmarks

Agent skill libraries decay quietly. An API migrates, a pinned dependency bumps a version, and the skill built against the old one keeps executing — misfiring in small ways until something downstream finally breaks. The real problem is granularity, not coverage: existing monitors flag whether an external value changed at all, not whether the skill actually depends on it. A version string sitting in a comment and the same string pinned inside a dependency spec look identical to a value-level monitor, but only one of them is an operational obligation. SkillGuard reframes this as skill drift-as-contract violation: it parses a skill document into environment contracts — typed records of which external values the skill's execution actually relies on — labels each operational or incidental, and checks only the operational ones against known drift events or live registry and URL evidence. Failed contracts double as repair instructions, pointing an LLM at exactly which assumption went stale rather than asking it to guess. The benchmark behind these numbers, DriftBench, releases 880 pairs: 174 constructed drift cases, 107 real drifts pulled straight from PyPI, npm, GitHub Actions, and Docker Hub changelogs, and 599 no-drift and hard-negative pairs built specifically to catch a monitor that over-fires on incidental change.

Value-level monitoring is the failure mode the paper measures directly: a contract-free CI probe that checks every extracted URL, version, and config value produces a 40% false-positive rate. Filtering by role eliminates it — SkillGuard raises zero false alarms across all 599 no-drift and hard-negative cases (Wilson 95% CI [0, 0.6]%), including 250 semantic hard negatives where the actual value changes (an alias URL, a non-breaking version bump) but the operational contract does not. On drift it's supposed to catch, the strongest backbone (Qwen3.6-Plus) hits 100% precision and 76% recall against known drift specs. In a pre-registered scan of 49 real, currently deployed skills, it surfaces live drift at 86% precision and 55% recall — flagging 14 skills, 12 of them genuine (the other 2 "false positives" were later adjudicated as real drift too). The repair payoff is the sharpest number here: telling a model only that a skill is stale fixes it in one round 10% of the time; handing it the specific violated contract instead gets that to 78%, at roughly half the token cost of a three-round self-refine loop that reaches a comparable 80%.

Key numbers

Contract-free CI probe false-positive rate	40%
SkillGuard false alarms over 599 no-drift/hard-negative cases	0% (Wilson 95% CI [0, 0.6%])
Known-drift precision / recall, best backbone	100% / 76%
Live discovery precision / recall, 49 real skills	86% / 55%
One-round repair success, no localization → contract-guided	10% → 78%

What this means for skill authors

For anyone authoring or maintaining a shared skill, the actionable move is documentation hygiene. SkillGuard's role classifier works because a well-written skill already signals which values are load-bearing — a pinned version inside a requirements block reads differently than the same string sitting in a changelog note. Skills that blur the distinction — unpinned examples sitting next to pinned dependencies, credentials described only in prose — give the classifier less to work with, and its own standalone audit shows the resulting gap: 89.5% precision but only 83.4% recall against human-labeled mentions, meaning some genuine obligations still get waved through as incidental. That argues for structure in the documentation itself, not just better monitors downstream. A catalog that scores whether a skill's environment assumptions are legible to automated checking — the way SkillFed's dual-tool scanning already looks for other structural signals — would catch a genuinely different failure mode than a security scan does.

The repair numbers are the more transferable lesson here. Detection alone fixes nothing — what changes the outcome is localization. Contract-guided repair reaches 78% one-round success mainly by turning a vague "something broke" into "this specific pinned value is stale, here's the evidence span," and it does that at roughly half the token cost of the multi-round self-refinement that lands at a similar 80%. Any skill-maintenance pipeline, federated catalog or single-agent memory, should treat a drift signal as incomplete unless it also names the exact assumption to fix. An alert without a location is barely better than no alert at all.

The paper doesn't oversell coverage, and that restraint holds up under scrutiny. Recall tops out at 76% even under known-drift conditions and drops to 55% in the wild; authentication and schema changes are the weak spot, with authentication drift caught only 67% of the time and repaired in none of the three test cases where it came up. Contract-based monitoring is pitched explicitly as complementary to execution-based canary testing, not a replacement for it — worth remembering before treating a clean scan as proof a skill still works.

Skills related to this research

playwright-verifier This skill automates accessibility compliance verification by testing whether your fixes actually resolve violations and don't create new ones. Built for Gemini, it provides structured testing workflows to catch accessibility regressions before they reach production.★ 370 Figma Generate Library Build production-ready design systems in Figma by orchestrating multi-phase workflows that extract tokens from code, create variable foundations, construct components with proper variant sets, and document design principles. Works alongside the figma-use skill to enforce quality patterns from real-world systems.★ 1,925 documentation-lookup Documentation-lookup pulls live reference material for popular frameworks and libraries when you need setup guidance, API details, or code examples. It resolves library identifiers and retrieves version-specific documentation to ensure your answers stay current and accurate.★ 5,778 codexer Codexer is a Python research assistant that leverages Context7 MCP servers to investigate libraries, fetch documentation, and evaluate dependencies for production use. It enforces strict coding standards including PEP 8 compliance, mandatory type hints, and modular architecture, with automated quality gates via black, flake8, and mypy. Use it to conduct thorough package research, compare solutions, and implement clean, reliable Python code.★ 6

Related notes

References

Fan, Tian, Li & Lu, "Skill Drift Is Contract Violation: Proactive Maintenance for LLM Agent Skill Libraries" (arXiv:2605.10990, 2026)
Han et al., "SWE-Skills-Bench: Do Agent Skills Actually Help in Real-World Software Engineering?" (arXiv:2603.15401, 2026)
Madaan et al., "Self-Refine: Iterative Refinement with Self-Feedback" (NeurIPS 2023)
Fruntke & Krinke, "Automatically Fixing Dependency Breaking Changes" (Proc. ACM on Software Engineering, FSE 2025)