+8.9 percentage points, p=0.71: Skills stall out when the tool layer already talks back

Notes on When Skills Don't Help: A Negative Result on Procedural Knowledge for Tool-Grounded Agents in Offensive Cybersecurity (arXiv:2605.20023) — Samuel Jacob Chacko, James Hugglestone, Chashi Mahiul Islam, Xiuwen Liu · May 2026

Note published July 25, 2026 · written by SkillFed’s research pipeline from the paper above · how these notes are made

This is a reanalysis, not a new experiment. It reuses a previously published 180-run study of an autonomous Capture-the-Flag agent — one grounded in the Model Context Protocol, so every tool call returns a schema-validated, low-latency observation — and reinterprets its four documentation conditions as a Skills ablation: a 591-token No-Skills baseline, a 12,865-token Experiential tier, a 17,253-token Curated tier, and a 36,001-token Comprehensive tier. Each tier ran against the same 15 challenges, three trials apiece, for 45 trials per condition. Read this way, the original study becomes exactly the controlled Skills-versus-no-Skills comparison that offensive-security benchmarks hadn't run before.

The lift is real but small, and it doesn't clear significance. Pass rate climbs from 77.8% with no documentation to 86.7% with the full Comprehensive tier, an 8.9-percentage-point gain. A chi-square test puts that at p=0.71, a Cochran–Armitage trend test at p=0.25, and five of six pairwise Cohen's h effect sizes land below the conventional small-effect threshold of 0.2. That's against a backdrop where SkillsBench's own cross-domain average is +16.2 percentage points over 84 tasks in 11 domains, with 16 of those tasks going negative. On one subtask, a timing side-channel exploit, more documentation actively hurt: Curated-Skills solved it 2 of 3 tries, Comprehensive-Skills only 1 of 3, which the authors attribute to false lesson propagation — written guidance overriding what the live tool feedback was already telling the agent. Their explanation is an environment-feedback bandwidth hypothesis: when the tool layer already returns strict, verifiable observations, that channel supplies the correction signal Skills exist to provide, so adding curated procedure on top is mostly redundant. When it conflicts with what the environment is already saying, it's actively harmful.