Agentic benchmarks 26 papers

Every note, newest paper first

1. 8/10 → 3/10: a 300K-character context collapses a code-audit skill's pass rate — relevant or not

   2026-07-20 — A fixed 24-check code-audit agent skill got stress-tested inside a production-derived auditing task, run through Codex with gpt-5.4-mini. The task and its verification checks stayed constant; only…

2. A 96,401-skill curated corpus lifts agent pass rates +7.5pp — until coverage runs out Bridge: benchmarks × retrieval

   2026-07-17 — SkillCorpus turns the sprawling public SKILL.md ecosystem into one deployable, licence-clean corpus. A six-stage pipeline parses, deduplicates, and quality-scores roughly 821,000 crawled skill…

3. Code review, testing, and security auditing claim 35% of task assignments; requirements analysis gets 2%

   2026-07-10 — The first large-scale empirical study of software-engineering skills starts with a brutal filtering funnel: 775,790 skills pulled from four public marketplaces — ClawHub, SkillHub, SkillNet, and…

4. Coding agents violate their own skill's embedded logic in up to 70% of test cases

   2026-07-10 — SkillLogic is a static-analysis framework that reads an agent skill file and extracts the logical relations binding its instructions together: preconditions that gate an action, postconditions…

5. LLM-generated skills move data-science accuracy 1.2 points — same as filler text

   2026-07-08 — The team built one reusable skill file per stage of a data-science agent's workflow — data preparation, data extraction, statistical analysis, and reporting — generating each with Gemini 2.5 Pro…

6. Rubric-filtered training lifts a 9B model to 32% accuracy — outcome-only filtering caps out at 18% Bridge: benchmarks × evolution

   2026-07-02 — SkillCoach splits agentic skill-use into four dimensions scored separately — skill selection, skill following, skill composition , and skill-grounded reflection — pulled from real agent rollouts…

7. SKILL.md files average 10.5 quality smells, and age doesn't cure them

   2026-07-01 — An Agent Skill 's domain knowledge lives inside SKILL.md — a mandatory file with metadata up top and then a markdown body authors can write however they want. No required sections, no schema for…

8. Usefulness Doesn't Predict Safety: 7.5% of Top Skills Fail the Scan Bridge: benchmarks × security

   2026-06-21 — SkillAudit swaps fixed-benchmark grading for skill-centered assessment : feed it an arbitrary skill package, it parses the SKILL.md, and generates capability-aligned tasks that probe exactly what…

9. Same skill, +22 points for Claude Sonnet, +5.5 for Nemotron Nano

   2026-06-16 — An automated pipeline turns any agent skill into a battery of executable tests. It reads the skill's content, provisions whatever the task needs — CLI tools, credentials, a specific runtime —…

10. Progressive Disclosure Triples Resource Touches — Pass Rate Moves Just 4 Points

    2026-06-10 — SkillJuror isolates Skill organization as a variable independent of content — a distinction most Agent Skill benchmarks conflate. It starts from a shared source Skill bundle and builds two…

11. Skills Pass the Task While Skipping 54-61% of Their Own Instructions

    2026-06-09 — Task success hides which parts of a skill an agent actually used. Skill coverage is a new test adequacy metric built to expose that: it parses each skill's natural-language instructions into…

12. SciVis agent skills swing task scores +5% to +66% — but bigger gains don't guarantee more tasks finish

    2026-06-04 — SciVisAgentSkills packages tool-specific procedural knowledge for four scientific visualization platforms — ParaView, napari, VMD, and TTK — into self-contained, version-pinned agent skills . Each…

13. 48% of the time, agents don't even read their own skill file

    2026-05-22 — OpenSkillEval builds its evaluation harness around real, evolving artifacts instead of a frozen benchmark: it pulls fresh presentation decks, web pages, posters, dashboards, and reports from the wild…

14. +8.9 percentage points, p=0.71: Skills stall out when the tool layer already talks back

    2026-05-19 — This is a reanalysis, not a new experiment. It reuses a previously published 180-run study of an autonomous Capture-the-Flag agent — one grounded in the Model Context Protocol , so every tool call…

15. Best skill-generation pipeline clears just 14% of code-repo tasks, 25% of document tasks Bridge: benchmarks × evolution

    2026-05-18 — SkillGenBench splits the skill lifecycle in two and scores only the first half: not whether an agent can use a skill, but whether a pipeline can generate one. A generator sees only raw source…

16. A fine-tuned 8B retriever hits 83 NDCG@10 — a 12B off-the-shelf model manages 55

    2026-05-07 — SkillRet turns skill retrieval into its own measurable problem, separate from end-to-end agent benchmarks. The corpus starts at 22,795 public agent skills scraped from claude-plugins.dev, then…

17. 71% of public healthcare skills carry no safety-boundary statement Bridge: benchmarks × security

    2026-05-04 — Researchers built the first systematic census of healthcare-focused agent skills — self-contained instruction packages that an agent loads only once a task matches the skill's stated description, a…

18. Only 0.1 points separate agents that need a skill from ones that don't Bridge: benchmarks × retrieval

    2026-04-27 — Cramming every skill's SKILL.md into context stops scaling long before a library gets large: context budgets fill up, and the calling model gets measurably worse at picking the right entry out of the…

19. 0.449 vs. 0.300: An Automated Skill Audit Out-Agreed Its Human Reviewers Bridge: benchmarks × security

    2026-04-22 — Medical-research agent skills carry a failure mode that general-purpose skill checks don't catch: a skill can run cleanly, pass every schema check, and still fabricate a citation or wander into…

20. Agents Writing Their Own Skills Close Just 45% of the Gap to Human Authors Bridge: benchmarks × evolution

    2026-04-22 — SkillLearnBench is the first benchmark built to test whether an agent can write its own agent skills well enough to matter, rather than testing skills a human already wrote. Researchers from…

21. Forced to search a 34k-skill library instead of being handed the right one, agents' pass rate falls from 55.4% to 38.4% -- a hair above the 35.4% no-skill floor Bridge: benchmarks × retrieval

    2026-04-06 — Liu et al. build a 34,198-skill test bed by pulling metadata from two community skill hubs, skillhub.club and skills.sh, then downloading the full SKILL.md folders from their GitHub sources --…

22. 534 of 3,984 public skills carried a critical security flaw — SkillTester grades against that baseline Bridge: benchmarks × security

    2026-03-28 — SkillTester is a comparative test harness for agent skills — the packaged, SKILL.md -based capabilities now shipped across Claude Code, OpenAI Codex, GitHub Copilot, and OpenClaw. Its core rule,…

23. 39 of 49 Real-World SWE Skills Move the Pass Rate by Exactly Zero

    2026-03-16 — SWE-Skills-Bench tests whether packaging procedural knowledge as an agent skill changes outcomes on real software engineering work, not on synthetic coding puzzles. It starts from a pool of over…

24. Flat Skill Invocation Craters From 48 to 17 as the Pool Hits 200,000 Bridge: benchmarks × retrieval

    2026-03-02 — AgentSkillOS tackles a scale problem: as of late February 2026, the Claude skill ecosystem held more than 280,000 public skills, built almost entirely by decentralized third-party authors, with no…

25. Curated Skills Add 16.6 Points — Self-Authored Ones Cost You 11.5

    2026-02-13 — SkillsBench runs Agent Skills — structured procedural packages loaded into an LLM agent's context at inference time — through a controlled, paired test instead of a single-condition demo. The…

26. 40,285 Skills Later, Supply Still Doesn't Match Demand

    2026-02-08 — Ling, Zhong, and Huang treat a live marketplace as a dataset, not a sample. All 40,285 publicly listed agent skill listings get pulled and measured — when they're published, what category they land…