Demystifying Agent Skills: Why They Work-Until They Don't

Skills in LLM agents don't work by injecting missing facts. They work by stabilizing action sequences the agent already knows how to execute but keeps fumbling. That's the central finding of this paper, and it reframes how the whole field should think about agent memory.

The mechanism is called procedural anchoring, and it accounts for 65.7% of observed skill effects across 528 matched trajectory triples. Explicit knowledge injection — the intuitive story about why skills help — accounts for just 4.5%. The difference matters enormously for how you design and evaluate skill systems.

The paper's methodology is worth understanding. Rather than reporting aggregate task success, the authors normalize 8,135 trial records, open-code 240 sampled trajectories, and build a 12-mode taxonomy validated against independent human annotation at 95.8% exact agreement. Each analysis unit is a triple: the same task run under raw execution, workflow memory injection, and skill injection. This lets them attribute behavioral changes to specific mechanisms rather than to outcome differences alone.

Skills beat workflow memory by 6.06 percentage points in matched comparisons, even though both are built from identical source trajectories. The gap traces to what workflow memory preserves: failed branches, verbose exploration, and process noise that inflate context and cause timeout. Workflow memory's timeout failure rate hits 10.6% versus 4.4% for skills. Skills compress the same experience into cleaner operational procedures — which setup steps to run, which tool sequences to follow, which pitfalls to avoid.

Retrieval is a separate and underappreciated failure mode. As skill pools grow from 5 to 100 candidates, actual-use precision collapses from 29.6% to 3.3% on average. Yet downstream task success barely moves, shifting from 36.4% to 39.3%. Exact ground-truth skill invocation is neither sufficient nor necessary for task completion. Semantically similar distractors are the dominant stressor for offline identification — embedding top-1 precision on similar pools falls from 70.5% to 53.4% as pool size grows — but execution success is largely insensitive to this degradation. Agents often retrieve related non-ground-truth skills that still provide partial procedural support.

Skills also introduce their own failure surface. The mode skill_guidance_misapplied_or_ignored appears in 10.0% of skill-arm cases versus 0.8% in raw execution. Compact abstractions require the agent to judge applicability, adapt conditions, and decide when to abandon guidance — and agents frequently get this wrong.

The paper is honest about scope: evaluation covers terminal and tool-use benchmarks only, a limited set of agent-model configurations, and a taxonomy induced from roughly 3% of normalized records. Rare behavioral modes may be underrepresented. But within those bounds, the controlled design is rigorous, and the findings are specific enough to act on.