48% of the time, agents don't even read their own skill file

Notes on OpenSkillEval: Automatically Auditing the Open Skill Ecosystem for LLM Agents (arXiv:2605.23657) — Jiahao Ying, Bo Ai, Wei Tang, Siyuan Liu, Yixin Cao · May 2026

Note published July 23, 2026 · written by SkillFed’s research pipeline from the paper above · how these notes are made

AI-assisted notes · reviewed by SkillFed Agentic benchmarks

OpenSkillEval builds its evaluation harness around real, evolving artifacts instead of a frozen benchmark: it pulls fresh presentation decks, web pages, posters, dashboards, and reports from the wild and turns each into a task an agent has to reproduce or extend. The benchmark spans five domains — slide generation, front-end web design, poster generation, data visualization, and report writing — yielding more than 600 task instances tested against 30 community skills pulled from repositories including clawhub.ai, skills.sh, openskills.space, and skillsmp.com. Each skill runs head-to-head against a no-skill baseline across a matrix of models and agent frameworks (Claude Code, Codex, and others), scored on both the execution trajectory — did the agent look at the skill, and when — and the quality of the final artifact.

The headline result is that having a skill and using a skill are different things. Left to their own devices, agents open the provided skill file in only about 48% of runs on average — and Claude Opus 4.6, one of the strongest models tested, reads it in only around 20% of cases. Forcing consultation pushes the read rate to 94% and moves first access earlier in the run, but that doesn't make skill augmentation free: runs with a skill attached burn 3-5x more tokens than the same task without one, and much of that spend isn't repaid in quality — plenty of widely-used skills failed to beat the no-skill baseline outright. The payoff also splits by task type. Skills packing rich design assets — templates, layout references — measurably lifted presentation and poster generation; front-end web design was already strong with no skill needed; reasoning-heavy work like data visualization and report generation saw only marginal gains from any skill tested. Claude Opus 4.6 running inside Claude Code and GPT-5.5 running inside Codex led overall; GPT-5.5 held a clear token-efficiency edge, while DeepSeek V4 offered the best open-weight cost-performance trade.

Key numbers

Skill file read rate (agent's own choice)	~48%
Claude Opus 4.6 read rate	~20%
Read rate when consultation is forced	94%
Token overhead running with a skill	3-5x
Task instances / skills evaluated	600+ / 30

What this means for skill authors

For skill authors, the read-rate numbers are the ones to sit with. A skill nobody opens is dead weight no matter how well it's written, and whether an agent reads the file at all depends as much on the framework and prompting setup as on the skill's own content. If a skill matters enough to ship, make invocation hard to skip — explicit read-then-act instructions, unambiguous triggering conditions. Sitting available in a directory is not the same as being used, even by a model as capable as Opus 4.6.

For skill selection, the 3-5x token overhead is the number to price in — it isn't automatically bought back in quality. Plenty of popular skills lost to the plain agent while still costing more to run, which means popularity and performance point in different directions often enough to matter. That's the exact gap a vetted catalog exists to close: checking a skill's actual lift on representative tasks instead of trusting download counts or stars. Dual-tool scanning aimed at that distinction — does a skill actually help, not just get installed — is qualitatively the discipline this ecosystem is missing.

For anyone building analytical skills — reporting, data visualization — the ceiling isn't formatting, it's reasoning. Design-asset-rich skills won on visual tasks while reasoning-oriented ones stalled, which points to where the next generation of report and analysis skills needs to improve: better decomposition and judgment scaffolding, not another template pack.

Skills related to this research

Figma Swiftui Figma Swiftui bridges design and development by translating between Figma files and SwiftUI code in both directions. Route design-to-code requests to build iOS interfaces from Figma frames, or code-to-design to push SwiftUI views and tokens back into Figma, preserving semantic colors, SF Symbols, and iOS patterns throughout.★ 1,925 Figma Use Figma Use is the prerequisite skill for running Plugin API code in Figma files. Load it before calling the `use_figma` tool to perform write operations, inspect file structure, manage variables and components, or execute custom logic. The skill provides critical rules for text editing, auto-layout, color ranges, and incremental workflows to avoid common failures.★ 1,925

Related notes

References

Jiahao Ying et al., "OpenSkillEval: Automatically Auditing the Open Skill Ecosystem for LLM Agents," arXiv:2605.23657 (2026)
Anthropic, "System card: Claude Opus 4.6"
"GPT-5.5 system card" (OpenAI)
Anthropic, "Equipping agents for the real world with agent skills"
"Codex by OpenAI | AI coding agent"