Agentic benchmarks 26 papers

Liu et al. build a 34,198-skill test bed by pulling metadata from two community skill hubs, skillhub.club and skills.sh, then downloading the full SKILL.md folders from their GitHub sources -- keeping only permissively licensed, well-formed, de-duplicated entries. Against that collection they run three models, Claude Opus 4.6, Kimi K2.5, and Qwen3.5-397B-A17B, through six settings of increasing realism on SkillsBench's 84 tasks (three runs each): skills hand-picked and forced into context, hand-picked without forcing, hand-picked with distractor skills mixed in, retrieved by the agent itself from the full 34k with the right skill still in the pool, retrieved with it removed from the pool, and a no-skill floor. A second track scores five retrieval strategies by recall@5 -- a single-shot dense-embedding lookup versus four agentic variants where the model itself issues search queries and judges what comes back -- and tests two ways to fix up whatever skill gets retrieved: query-specific refinement , which adapts the skill to the task at hand, and query-agnostic refinement , a one-time edit made with no particular task in view.

SkillTester is a comparative test harness for agent skills — the packaged, SKILL.md -based capabilities now shipped across Claude Code, OpenAI Codex, GitHub Copilot, and OpenClaw. Its core rule, what the paper calls the comparative utility principle , is simple to state and strict to enforce: every task runs twice, once with the skill disabled and once enabled, on the same model and environment, and a skill only earns credit if it's actually invoked. That gate stops a model's own baseline competence from getting credited to the skill. Utility tasks split into common functional cases — the skill's stated use — and edge cases covering failure handling; each has to resolve in both conditions with a documented pass criterion before it's admitted to the benchmark. Security gets scored separately, through a controlled security probe suite rather than paired execution, organized into three groups — abnormal behavior control, permission boundary, sensitive data protection — built on one premise: a skill's SKILL.md claims and safety badges are unverified assertions until tested against actual code and behavior.

SWE-Skills-Bench tests whether packaging procedural knowledge as an agent skill changes outcomes on real software engineering work, not on synthetic coding puzzles. It starts from a pool of over 84,000 public skills and filters down to 49 spanning six SWE subdomains — deployment and DevOps, analytics and monitoring, API development, data science and ML, security and testing, and developer tools — each paired with an authentic GitHub repo pinned at a fixed commit plus a requirement document carrying explicit acceptance criteria. A requirement-driven verification framework turns each acceptance criterion into an executable pytest check, so every task instance (about 565 total, roughly 11 per skill) gets a deterministic pass/fail verdict instead of a judged score. Every task runs twice — once with the skill's SKILL.md dropped into the agent's context, once without — under a single fixed agent configuration (Claude Code driving Claude Haiku 4.5), so the skill's presence is the only thing that changes.

AgentSkillOS tackles a scale problem: as of late February 2026, the Claude skill ecosystem held more than 280,000 public skills, built almost entirely by decentralized third-party authors, with no structured way for a user or agent to browse that space. The fix runs in two stages. Offline, a capability tree organizes the pool through recursive LLM categorization — each node splits into child categories once its skill count exceeds a per-node capacity, collapsing even a 200,000-skill ecosystem into a shallow, browsable hierarchy. Past a size threshold, only the top-installed skills stay in this active tree (10,000 of 200,000 at the largest scale tested); the rest sit in a dormant, embedding-searchable index. At task time, the system walks the tree to shortlist candidates, prunes down to the top eight, and compiles them into a DAG-based orchestration plan under one of three named strategies (Quality-First, Efficiency-First, Simplicity-First) before executing skills in dependency order. Testing this against the obvious alternative — handing an agent the same skills unstructured — required a benchmark built for the purpose: 30 tasks spanning five artifact categories (data computation, document creation, motion video, visual design, web interaction), graded by pairwise LLM judging distilled into a Bradley-Terry model score.

SkillsBench runs Agent Skills — structured procedural packages loaded into an LLM agent's context at inference time — through a controlled, paired test instead of a single-condition demo. The benchmark assembles 87 tasks across 8 domains (software engineering, natural science, cybersecurity, finance, and more), each shipped with a curated Skill and a deterministic verifier so pass/fail isn't a judgment call. The task pool itself passed through selection: 400 candidate submissions from 142 contributors, an automated gate (structural integrity, oracle execution, AI-text detection, leakage checks), then human review of at least 30 minutes per task, for a 22% acceptance rate. Every task runs three trials under matched no-Skills and curated-Skills conditions, across 18 model- harness combinations spanning OpenHands, Claude Code, Gemini CLI, and Codex CLI paired with models like GPT-5.5, Gemini 3.1 Pro, GLM 5.1, and MiniMax M2.7.

Ling, Zhong, and Huang treat a live marketplace as a dataset, not a sample. All 40,285 publicly listed agent skill listings get pulled and measured — when they're published, what category they land in, how long they run, how much they're actually used, how similar they are to each other, and what actions they're allowed to take. The payoff is a corpus-scale baseline for what an agent skill ecosystem looks like in practice, not aspiration.