Same skill, +22 points for Claude Sonnet, +5.5 for Nemotron Nano

Notes on A Framework for Evaluating Agentic Skills at Scale (arXiv:2606.17819) — Maksim Shaposhnikov, Nicolas Fortuin, Simon Stipcich, Maria I. Gorinova, Amy Heineike, Robert S. Willoughby · June 2026

Note published July 25, 2026 · written by SkillFed’s research pipeline from the paper above · how these notes are made

An automated pipeline turns any agent skill into a battery of executable tests. It reads the skill's content, provisions whatever the task needs — CLI tools, credentials, a specific runtime — drafts a realistic task description, and writes two hidden scoring rubrics, then runs a validation pass to confirm the task doesn't leak its own answer. Applied to 500 real-world skills pulled from public registries and sourced from roughly 100 organizations, including Anthropic, Google, ElevenLabs, and Shopify, the pipeline generated about 1,000 tasks. Each one was solved twice — once with the skill available, once without — across 19 proprietary and open-source model configurations spanning Claude, GPT, Gemini, GLM, DeepSeek, Kimi, MiniMax, Qwen, and Nemotron. That produced roughly 38,000 scored trajectories, graded on two separate rubrics: did the output actually work (goal completion), and did it follow the skill's encoded conventions (instruction-following).

Skill access reliably moves the needle, but by wildly different amounts. The aggregate score gain — the skill delta — ranges from +5.5 points for Nemotron 3 Nano 30B up to +22.1 for Claude Sonnet 4.6, and it comes almost entirely from instruction-following: goal completion was already high without a skill and climbs above 90% for most models once the skill is present, Nemotron being the one holdout. Frontier instruction-following scores cluster in the mid-to-high 80s (Opus 4.8 at 88.0, Opus 4.7 at 87.7, the open-weights GLM 5.1 at 85.0), while Kimi K2.6, MiniMax 2.7, and Qwen3-Coder-Next sit 25-30 points lower even with the skill in hand. Give a cheaper model the skill and it closes in on its flagship sibling: GLM 5.1 hits an overall score of 91.1, within 1.2-1.6 points of Opus 4.7's 92.3 and Opus 4.8's 92.7, at roughly a quarter of the per-scenario cost. The uplift is domain-dependent too — skills that encode concrete step-by-step workflows, like media file processing or security checklists, gain over +30 points, while skills that mainly state general best practices, like testing or scientific computing, gain closer to +17.