8/10 → 3/10: a 300K-character context collapses a code-audit skill's pass rate — relevant or not

Notes on How Agent Skills Fail under Long Contexts: A White-Box Study in Code Auditing (arXiv:2607.17937) — Yue Xue · July 2026

Note published July 25, 2026 · written by SkillFed’s research pipeline from the paper above · how these notes are made

A fixed 24-check code-audit agent skill got stress-tested inside a production-derived auditing task, run through Codex with gpt-5.4-mini. The task and its verification checks stayed constant; only the surrounding context changed — a clean ~11,000-character run, a topically relevant ~299,000-character run, and an irrelevant ~299,000-character run padded to match. Ten runs per condition, and every failure was hand-classified into one of four buckets: a requirement dropping out of the active context, a requirement edited away mid-trajectory, a violation the agent noticed but never fixed, or a harness/evaluator failure unrelated to the agent.

Pass rate fell from 8/10 in the clean run to 3/10 in both long conditions — the same outcome whether the padding was relevant or noise, which points at context volume rather than topical distraction as the driver. That's a 50-point gap, but with only ten runs per condition it lands at trend-level significance under a two-sided Fisher's exact test (p = 0.0698). The odder result sits underneath it: requirement coverage held above 92% in both failing long-context conditions. Most individual checks still passed — a handful of dropped requirements was enough to sink the whole artifact. A second audit task passed every clean and long run, so this isn't a universal context-length cliff, and swapping a generic self-check prompt (5/10) for a detailed, itemized external checklist (10/10, p = 0.0325) restored full reliability.