Skill shadowing — not context bloat — drives up to 68% of a 202-skill library's 21% pass-rate drop

Notes on More Skills, Worse Agents? Skill Shadowing Degrades Performance When Expanding Skill Libraries (arXiv:2605.24050) — H. Song, Songtao Wei · May 2026

Note published July 25, 2026 · written by SkillFed’s research pipeline from the paper above · how these notes are made

AI-assisted notes · reviewed by SkillFed Skill retrieval

A skill library doesn't have to get big before it starts working against you. Two Databricks researchers pin down exactly when: past the handful of skills a task actually needs, agents get measurably worse. They define an oracle skill set for each task — the skill or skills that, tested alone, lift pass rate by at least 4 percentage points over no skill at all — then track the pass-rate gap between an agent given only that oracle set and the same agent buried in the full library. A probability identity over which skills the agent actually invokes splits that gap cleanly into two effects: skill shadowing, where the agent picks the wrong skill more often as the library expands, and context overhead, where a longer list of skill descriptions hurts execution even when selection is right. They bound both effects mathematically and test the split on SkillsBench, running Claude Haiku 4.5 and Sonnet 4.6 agents against libraries of 52, 102, and 202 skills across 38 (task, model) pairs and 2,545 trajectories.

Pass rate drops as much as 21% between the oracle-only baseline and the 202-skill library — and shadowing accounts for nearly all of it. The shadowing gap widens from 3 points of drop at 52 skills to 14 points at 202, up to 68% of the total decline, and it's the only one of the two effects whose confidence interval ever clears zero. Context overhead just sits there: a consistently positive but statistically indistinguishable-from-zero 6 to 7 points, at every library size. You can see the mechanism directly in the invocation data. Oracle-only invocations fall from 88% to 52.6% of trajectories as the library grows to 202 skills, and the lost ground splits between agents abandoning skill use entirely (12% to 38.5% of trajectories) and agents invoking a wrong or mixed set of skills. The clearest single case: asked to count coins in Mario footage, a distractor skill named "video-frame-extraction" out-competed the correct skill's description in all 26 of 26 recorded trajectories.

Key numbers

Pass-rate drop, oracle set → 202-skill library	up to 21%
Share of the drop from skill shadowing	up to 68%
Oracle-only invocations at 202 skills (from 88%)	52.6%
Trajectories evaluated	2,545
Wrong-skill invocation rate, Mario coin-counting task	26 of 26 trajectories

What this means for skill authors

Selection happens on name and description alone — the body of a skill stays invisible until it's already won the competition for invocation. That means a well-built skill can lose to a worse one that simply describes itself with more confidence. The failure is silent, too: nothing errors, the agent just runs the wrong instructions with full authority. Past a few dozen skills, description disambiguation stops being a nice-to-have. It's the main lever available before an agent ever executes a task.

Haiku 4.5 and Sonnet 4.6 fail in different ways, and that matters for anyone debugging agent behavior. Haiku's degradation shows up earlier and leans toward giving up — reverting to no skill at all rather than guessing wrong. Sonnet holds up longer, but when it breaks, it tends to invoke a wrong or mixed set of skills — a failure mode indistinguishable from normal operation unless you check which skill actually fired. A pass/fail eval won't catch that. You need the invocation trace.

This is really an argument for curating what an agent even sees, not just for writing better skill descriptions. SkillFed's dual-tool scanning and vetted catalog exist to prune overlapping or redundant skill entries before they ever compete for an agent's attention — the same instinct the paper's authors apply by hand, deduplicating and disambiguating the 88 authored bundles that make up their largest test library. Shadowing scales with how many similar-sounding options sit in front of the agent, which is exactly why cutting the candidate set beats trying to out-write every distractor.

Skills related to this research

Figma Generate Library Build production-ready design systems in Figma by orchestrating multi-phase workflows that extract tokens from code, create variable foundations, construct components with proper variant sets, and document design principles. Works alongside the figma-use skill to enforce quality patterns from real-world systems.★ 1,925 documentation-lookup Documentation-lookup pulls live reference material for popular frameworks and libraries when you need setup guidance, API details, or code examples. It resolves library identifiers and retrieves version-specific documentation to ensure your answers stay current and accurate.★ 5,778 codexer Codexer is a Python research assistant that leverages Context7 MCP servers to investigate libraries, fetch documentation, and evaluate dependencies for production use. It enforces strict coding standards including PEP 8 compliance, mandatory type hints, and modular architecture, with automated quality gates via black, flake8, and mypy. Use it to conduct thorough package research, compare solutions, and implement clean, reliable Python code.★ 6 excalidraw-diagram-generator Turn natural language into Excalidraw diagrams without drawing manually. Supports flowcharts, mind maps, architecture diagrams, sequence diagrams, ER diagrams, and data flow visualizations. The skill analyzes your description, identifies key elements and relationships, then generates a ready-to-open .excalidraw file.★ 6

Related notes

References

H. Song and S. Wei (2026). More Skills, Worse Agents? Skill Shadowing Degrades Performance When Expanding Skill Libraries. arXiv:2605.24050.
X. Li et al. (2026). SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks. arXiv:2602.12670.
T. Gan and Q. Sun (2025). RAG-MCP: Mitigating Prompt Bloat in LLM Tool Selection via Retrieval-Augmented Generation. arXiv:2505.03275.
R. Xu and Y. Yan (2026). Agent Skills for Large Language Models: Architecture, Acquisition, Security, and the Path Forward. arXiv:2602.12430.
Y. Liu et al. (2026). How Well Do Agentic Skills Work in the Wild: Benchmarking LLM Skill Usage in Realistic Settings. arXiv:2604.04323.