Agentic benchmarks 26 papers

A fixed 24-check code-audit agent skill got stress-tested inside a production-derived auditing task, run through Codex with gpt-5.4-mini. The task and its verification checks stayed constant; only the surrounding context changed — a clean ~11,000-character run, a topically relevant ~299,000-character run, and an irrelevant ~299,000-character run padded to match. Ten runs per condition, and every failure was hand-classified into one of four buckets: a requirement dropping out of the active context, a requirement edited away mid-trajectory, a violation the agent noticed but never fixed, or a harness/evaluator failure unrelated to the agent.

SkillCorpus turns the sprawling public SKILL.md ecosystem into one deployable, licence-clean corpus. A six-stage pipeline parses, deduplicates, and quality-scores roughly 821,000 crawled skill files down to 96,401 survivors: an exact-match tier collapses 169,465 duplicates outright, a semantic tier merges near-duplicates above a cosine-similarity threshold and sends borderline pairs to an LLM judge, and a final safety hard-gate plus OSI-licence filter removes skills flagged for prompt injection, command injection, unsafe execution, or an unclear licence. Every surviving skill gets one of 16 task-class labels and three independent quality scores — utility, robustness, and safety — instead of one blended number, and the corpus ships with a fine-tuned retrieval-and-selection stack : an embedding recall model, a reranker, and an LLM gate that reads each candidate's full body before injecting up to two skills into an agent's prompt.

The first large-scale empirical study of software-engineering skills starts with a brutal filtering funnel: 775,790 skills pulled from four public marketplaces — ClawHub, SkillHub, SkillNet, and SkillsMP — shrink to 11,497 after deduplication, rule-based and LLM-based relevance filtering, and dead-link removal. Every surviving skill gets mapped onto an eight-stage SE lifecycle taxonomy running from requirements through release, then broken down further by activity, measured for length and internal structure (plain instructions versus scripts versus full agent workflows), version history, and — wherever a marketplace exposes one — its own safety or quality score.

SkillLogic is a static-analysis framework that reads an agent skill file and extracts the logical relations binding its instructions together: preconditions that gate an action, postconditions that stay unmet until a follow-up step runs, constraints that narrow how an allowed action can be carried out, fallbacks that only kick in after a primary route fails, plus conjunctions, exceptions, overrides, and conflicts — eight relation types in all. Run against 5,224 public skills pulled from SkillsMP, the scanner found that 70% carry at least one such relation — most skills already demand that an agent reason about sequencing and precedence, not just execute a single command. The highest-confidence, highest-impact, locally-gradable relations feed into SLBench: 86 audited cases, each built around a seeded repository and a grader that checks execution artifacts — files written, configs changed, logs left behind — rather than the agent's own account of what happened.

The team built one reusable skill file per stage of a data-science agent's workflow — data preparation, data extraction, statistical analysis, and reporting — generating each with Gemini 2.5 Pro and shipping it unedited, mirroring how a low-curation skill pipeline actually gets used. Each skill split into four parts: routing triggers, a core procedure, worked examples, and reference notes. Rather than testing skills as a single on/off switch, they ran a component ablation : five conditions from no-skill up to the full four-part file, plus a token-matched control of irrelevant office-supply text padded to the same length, and a variant instructing the model to prioritize task instructions over skill guidance. Nine model configurations across OpenAI, Google, and Anthropic — spanning compact, frontier, and explicit-reasoning tiers — ran three times each over 56 deterministically verified tasks, yielding 7,560 runs in the main ablation and 10,584 runs total with the supplemental controls.

SkillCoach splits agentic skill-use into four dimensions scored separately — skill selection, skill following, skill composition , and skill-grounded reflection — pulled from real agent rollouts instead of fixed up front. Each task gets its own self-evolving rubric : a first draft built from the gold skill's steps, then reworked across rounds of judging real trajectories, proposing local patches, and keeping only the ones that clear a validation gate on held-out data. That rubric score runs alongside the task's external pass/fail verifier, never inside it — a trajectory that stumbles into the right answer gets scored differently from one that followed the skill as written. To stress-test selection, agents face a distractor-augmented skill library : gold skills buried among overlapping, superficially similar decoys meant to mimic a crowded enterprise repository.

An Agent Skill 's domain knowledge lives inside SKILL.md — a mandatory file with metadata up top and then a markdown body authors can write however they want. No required sections, no schema for the body. This study is the first systematic look at what actually ends up in that free space. The authors pulled 238 real skills from the skills.sh marketplace, working down from 133,149 listed packages by filtering on weekly downloads and repository diversity, then hand-coded the bodies into a taxonomy of 13 higher-level and 44 lower-level semantic components — task, introduction, references, principles, and so on. To judge quality, they ran a multivocal literature review across 29 online sources — docs, vendor guides, practitioner posts — and distilled 26 concrete authoring best practices. Violating each one gets a name: a skill smell . A hybrid detector, static rules for some smells and an LLM classifier for the rest, then scanned the corpus and tracked 142 skills with commit history (1,199 commits total) to see whether smells clear up as a skill matures.

SkillAudit swaps fixed-benchmark grading for skill-centered assessment : feed it an arbitrary skill package, it parses the SKILL.md, and generates capability-aligned tasks that probe exactly what the skill claims to do. Each task runs twice in an isolated sandbox — once with the skill installed, once without — same backbone model, same configuration, same rubric. That pairing is the baseline comparison principle : it isolates what the skill actually contributes from what the backbone could already do alone. A separate two-stage safety pipeline runs in parallel. An LLM first scans the package against 21 predefined risk patterns across five categories; every flagged risk then becomes a live runtime probe — an agent actually attempting the risky action — and only a confirmed trajectory, file change, or network call counts against the skill.

An automated pipeline turns any agent skill into a battery of executable tests. It reads the skill's content, provisions whatever the task needs — CLI tools, credentials, a specific runtime — drafts a realistic task description, and writes two hidden scoring rubrics, then runs a validation pass to confirm the task doesn't leak its own answer. Applied to 500 real-world skills pulled from public registries and sourced from roughly 100 organizations, including Anthropic, Google, ElevenLabs, and Shopify, the pipeline generated about 1,000 tasks. Each one was solved twice — once with the skill available, once without — across 19 proprietary and open-source model configurations spanning Claude, GPT, Gemini, GLM, DeepSeek, Kimi, MiniMax, Qwen, and Nemotron. That produced roughly 38,000 scored trajectories, graded on two separate rubrics: did the output actually work ( goal completion ), and did it follow the skill's encoded conventions ( instruction-following ).

SkillJuror isolates Skill organization as a variable independent of content — a distinction most Agent Skill benchmarks conflate. It starts from a shared source Skill bundle and builds two knowledge-matched variants: a flattened Baseline that keeps every command, threshold, and helper contract in one file, and a Progressive Disclosure (PD) version that reorganizes the same content behind a short root SKILL.md pointing to on-demand references. A three-tier construction audit — deterministic file and path gating, an LLM rubric check across 968 items, and human adjudication on the 3 that failed — confirms the pair differs only in layout, not in task knowledge. Both variants then run through the same Codex/GPT-5.4 harness across 82 SkillsBench tasks, five trials per condition, 1,230 trials total, with every trajectory logged for downstream analysis.

Task success hides which parts of a skill an agent actually used. Skill coverage is a new test adequacy metric built to expose that: it parses each skill's natural-language instructions into semi-structured Skill Behavior Constraints (SBCs) — condition/behavior pairs in an EARS-style "when X, the agent shall Y" form — then checks every agent trajectory against every constraint, labeling each one "not covered," "pass," or "fail" from observable evidence: messages, tool calls, file state. Applied to SkillsBench , the framework extracted roughly 4,283 constraints across 202 task-linked skills. An LLM judge (DeepSeek V4 Flash) had to earn its keep against human annotators before being trusted at scale: 88.58% agreement on whether a constraint was covered at all, 81.89% on the full three-way label, against a human-human ceiling of 98.05% (κ=0.961) — proof the labeling rubric itself is reproducible, not just the judge.

SciVisAgentSkills packages tool-specific procedural knowledge for four scientific visualization platforms — ParaView, napari, VMD, and TTK — into self-contained, version-pinned agent skills . Each one pins software versions, distills official docs into working usage patterns, and borrows code straight from existing domain agents (ParaView-MCP, BioImage-Agent, GMX-VMD-MCP, TopoPilot). Failure-mode fixes get folded in too — headless rendering, viewport capture, the stuff that trips agents up in practice. The team ran these skills across two coding-agent harnesses — Claude Code on Sonnet-4.5, Codex on GPT-5.2 — against SciVisAgentBench : 108 expert-designed multi-step tasks across five suites — ParaView, molecular visualization, bioimage, topology, and object identification. Three trials per configuration, scored by a multimodal LLM judge (Claude Opus 4.6) plus deterministic completion checks and, for ParaView, image-similarity metrics.

OpenSkillEval builds its evaluation harness around real, evolving artifacts instead of a frozen benchmark: it pulls fresh presentation decks, web pages, posters, dashboards, and reports from the wild and turns each into a task an agent has to reproduce or extend. The benchmark spans five domains — slide generation, front-end web design, poster generation, data visualization, and report writing — yielding more than 600 task instances tested against 30 community skills pulled from repositories including clawhub.ai, skills.sh, openskills.space, and skillsmp.com. Each skill runs head-to-head against a no-skill baseline across a matrix of models and agent frameworks (Claude Code, Codex, and others), scored on both the execution trajectory — did the agent look at the skill, and when — and the quality of the final artifact.

This is a reanalysis, not a new experiment. It reuses a previously published 180-run study of an autonomous Capture-the-Flag agent — one grounded in the Model Context Protocol , so every tool call returns a schema-validated, low-latency observation — and reinterprets its four documentation conditions as a Skills ablation: a 591-token No-Skills baseline, a 12,865-token Experiential tier, a 17,253-token Curated tier, and a 36,001-token Comprehensive tier. Each tier ran against the same 15 challenges, three trials apiece, for 45 trials per condition. Read this way, the original study becomes exactly the controlled Skills-versus-no-Skills comparison that offensive-security benchmarks hadn't run before.

SkillGenBench splits the skill lifecycle in two and scores only the first half: not whether an agent can use a skill, but whether a pipeline can generate one. A generator sees only raw source material — a pinned code-repository snapshot or a long-form document bundle — and has to produce a standalone SKILL.md package. A separate, fixed executor (MiniMax-2.5) then runs that package against hidden, deterministic test cases, so generation quality is measured apart from prompting skill or task planning. The benchmark crosses two source types — repository-grounded, where the procedure is scattered across code, config, and scripts, and document-grounded, where it's explicit but spread across a manual or spec — with two regimes: task-conditioned generation, where the task is revealed before the skill is written, and task-agnostic generation , where a reusable skill library has to be distilled before any task is known. The 187 accepted tasks went through a five-stage pipeline — knowledge-graph extraction, scenario generation, test-case generation, a difficulty filter that throws out anything solvable without the corpus or trivially solvable with it, and reference-skill verification — plus human review. Five generation methods, from naive prompting to workflow-based and self-evolving pipelines, each paired with six backbone models — Claude Sonnet 4.5, GPT-5, Kimi K2.5, GLM-5, MiniMax-M2.7, and Qwen3.6-Plus — and scored on pass@3 .

SkillRet turns skill retrieval into its own measurable problem, separate from end-to-end agent benchmarks. The corpus starts at 22,795 public agent skills scraped from claude-plugins.dev, then narrows through five filtering passes — language, license (MIT/Apache-2.0 only), two rounds of deduplication — down to 17,810 skills, each tagged into a two-level taxonomy of 6 major and 18 sub-categories. These are long documents, not short tool blurbs: median length is 1,583 tokens, an order of magnitude past typical tool-retrieval benchmarks. Training queries (63,259) and evaluation queries (4,997) were generated separately over disjoint skill pools — Qwen3.5-122B-A10B for training, Claude Opus 4.6 for evaluation — then filtered for skill-name leakage and checked by both an LLM panel and human annotators. Just over half the evaluation queries name two or three skills at once, not one.

Researchers built the first systematic census of healthcare-focused agent skills — self-contained instruction packages that an agent loads only once a task matches the skill's stated description, a pattern the spec calls progressive disclosure . Starting from 58,159 public skills on ClawHub (an April 2026 snapshot), the team used an LLM classifier to pull 557 healthcare-related skills, then had a second LLM read each skill's full SKILL.md file to annotate ten dimensions covering function, care-cycle stage, intended user, input modality, autonomy level , and clinical decision impact .

Cramming every skill's SKILL.md into context stops scaling long before a library gets large: context budgets fill up, and the calling model gets measurably worse at picking the right entry out of the pile. Skill Retrieval Augmentation (SRA) splits that into three separable stages instead of one context-stuffing problem — retrieve candidates from a large external corpus, decide which of them actually make it into the active context ( skill incorporation ), then apply whatever got loaded to solve the task — and SRA-Bench grades each stage on its own rather than scoring pass/fail on the end result. The benchmark draws 5,400 test instances from six existing datasets spanning theorem proving, formal logic, tool-use QA, medical calculation, competition math, and code generation, each hand-annotated against one or more of 636 gold skills. Those gold skills sit buried inside 25,626 web-scraped distractor skills, producing a 26,262-entry corpus where only 2.4% of anything is actually relevant to a given query. Eight models — six open-weight (Qwen3-4B/32B/235B, Llama-3.1-8B, Llama-3.3-70B, Mistral-Small-3.1-24B) plus frontier GLM-5.1 and GPT-5.4 — ran across BM25, dense, and reranking retrieval, and three skill-use strategies: full injection of the single top-ranked result, LLM-driven selection from a top-50 shortlist, and an OpenClaw-style progressive disclosure setup where the agent pulls full skill content on demand instead of all at once.

Medical-research agent skills carry a failure mode that general-purpose skill checks don't catch: a skill can run cleanly, pass every schema check, and still fabricate a citation or wander into diagnostic territory it has no business entering. MedSkillAudit is a two-gate pre-deployment audit built to catch exactly that, run before a skill ever ships. A structural veto gate checks crash rate, schema compliance, result determinism, and code-injection surface; a domain-specific research veto gate checks for fabricated citations or data, practice-boundary violations, methodological fallacies, and code usability. A final score blends a 25-criterion static check (weighted 0.4) with a dynamic execution rubric (weighted 0.6), sorting each skill into one of four release tiers: Production Ready, Limited Release, Beta Only, Reject. The team ran it against 75 real medical-research skills — 15 each across five categories: Evidence Insight, Protocol Design, Data Analysis, Academic Writing, and a catch-all Other. Two experts then independently scored every skill on the same 0-100 scale and disposition ladder, which let the researchers measure two things at once: how closely the system's verdicts tracked human consensus, and how closely the two humans tracked each other.

SkillLearnBench is the first benchmark built to test whether an agent can write its own agent skills well enough to matter, rather than testing skills a human already wrote. Researchers from Carnegie Mellon and Amazon AGI assembled 20 verified tasks across 15 sub-domains — software engineering, information retrieval, productivity tools, data analytics, content creation, and utilities — pulled from a community skill-usage taxonomy. Every task is checked twice before it counts: an agent with no skill must fail it at least half the time over ten tries, and a human-authored reference skill must be able to solve it. Four continual learning methods for generating skills from a task description are then pitted against each other: one-shot generation, self-feedback (the agent critiques and rewrites its own skill after one failed attempt), teacher feedback (an outside evaluator gives guidance across up to two QA rounds), and Anthropic's structured Skill-Creator pipeline. Six different skill-writing LLMs do the generating — three Claude models, three Gemini models — while Claude Sonnet 4.6 stays fixed as the agent that actually executes whatever skill comes out.