Skill folders, not weights: +12.1 points on noisy-retrieval QA

Notes on EvoSkill: Automated Skill Discovery for Multi-Agent Systems (arXiv:2603.02766) — Salaheddin Alzubi, N. Provenzano, Jaydon Bingham, Weiyuan Chen, Tu Vu · March 2026

Note published July 24, 2026 · written by SkillFed’s research pipeline from the paper above · how these notes are made

AI-assisted notes · reviewed by SkillFed Skill evolution

EvoSkill discovers and edits agent skills — structured folders of instructions and helper scripts — by watching a coding agent fail, then testing fixes at the skill level instead of touching prompts or model weights. The system splits into three roles: an executor runs tasks under the current agent program, a proposer diagnoses the failures and drafts skill edits, and a skill-builder writes the result into a reusable folder. Whether an edit survives comes down to a Pareto frontier of three competing agent programs — a candidate only replaces the frontier's weakest member if it beats it on a held-out validation split the proposer never saw. The base model, Claude Code running Opus 4.5, never changes; every gain has to come from the skill folder, not the weights.

On OfficeQA, a Treasury-document reasoning benchmark, evolved skills push exact-match accuracy from 60.6% to 67.9%, using well under a fifth of the question set for training. The gain is bigger on SealQA, where web search returns noisy or conflicting results: 26.6% to 38.7%. What happens next is the sharper result. A search-persistence skill built entirely on SealQA's failures, one that enforces a rule of three independent sources before accepting an answer, drops unmodified into BrowseComp and still adds 5.3 points (43.5% to 48.8%) — no retraining, no BrowseComp-specific edits. That's evidence the skill captured a transferable search discipline, not a benchmark-specific trick.

Key numbers

OfficeQA exact-match accuracy (skill-merge config)	60.6% → 67.9% (+7.3 pts)
SealQA accuracy under noisy retrieval	26.6% → 38.7% (+12.1 pts)
Zero-shot transfer of SealQA skill to BrowseComp	43.5% → 48.8% (+5.3 pts)
Training data needed for OfficeQA gains	≤15% of 246 questions

What this means for skill authors

The frozen-model constraint is the paper's most important design choice for anyone building or consuming a skill catalog. A capability gain that survives without touching weights lives entirely in an artifact you can read, diff, and move between models — the whole premise of a portable skill format. In effect, EvoSkill automates what a careful skill author already does by hand: watch where an agent trips, write down the missing procedure, then check that the fix doesn't just patch the failures you happened to see.

The zero-shot transfer result is the one worth sitting with. A skill optimized against SealQA's noisy retrieval generalized to BrowseComp, a benchmark it never trained on. That's a useful triage question for a skill catalog: a skill that only helps on the exact failures used to build it is glue, not a capability. One built against a held-out gate that catches overfitting to those failures has a better shot at working on tasks nobody wrote it for. Vetting a submitted skill means asking the same question EvoSkill's frontier asks automatically — does this still help on cases it wasn't tuned against?

None of that substitutes for reading what a skill actually does before trusting it. EvoSkill's selection loop is a performance filter, not a safety one; a skill folder that scores well on held-out accuracy can still ship code nobody has inspected. Propose an edit, test it against unseen cases, keep a record of what was tried and why — that discipline is a fair template for how any vetting process, automated scanning included, should treat a skill update before promoting it into a catalog.

Skills related to this research

ultraqa UltraQA automates quality assurance by cycling through test execution, failure diagnosis, and targeted fixes until your specified goal succeeds. It supports tests, builds, linting, type checking, or custom verification patterns, with a maximum of five cycles and early exit on repeated failures. The workflow coordinates diagnosis and remediation across multiple agents to streamline the path to passing quality gates.★ 38,122 Ios Mcp Routing This skill enables iOS applications to efficiently route Model Context Protocol (MCP) messages and coordinate request handling across device components. It provides the infrastructure needed to manage bidirectional communication patterns and ensure reliable message delivery in mobile environments.★ 24 statistical-testing Statistical Testing provides researchers with advanced methods for hypothesis testing, Bayesian inference, survival analysis, time series modeling, and meta-analysis. The skill covers multiple comparison corrections, effect size calculations, and bootstrap/permutation approaches with APA-compliant reporting standards.★ 869 voice-localization This skill guides you through scaling video and audio content to global audiences using AI voice synthesis that preserves your brand character across languages. It provides decision frameworks for choosing between AI localization, traditional dubbing, and subtitles based on your content type and budget, plus production workflows that handle translation, voice generation, and quality assurance per market. Use it to expand into new language markets efficiently while keeping the same perceived voice speaking natively in each language.★ 140

Related notes

References

Alzubi, S., Provenzano, N., Bingham, J., Chen, W., & Vu, T. (2026). EvoSkill: Automated Skill Discovery for Multi-Agent Systems. arXiv:2603.02766.
Wang, G., Xie, Y., Jiang, Y., Mandlekar, A., Xiao, C., Zhu, Y., Fan, L., & Anandkumar, A. (2023). Voyager: An Open-Ended Embodied Agent with Large Language Models. arXiv:2305.16291.
Agrawal, L. A. et al. (2026). GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning. arXiv:2507.19457.
Novikov, A. et al. (2025). AlphaEvolve: A Coding Agent for Scientific and Algorithmic Discovery. arXiv:2506.13131.
Pham, T., Nguyen, N., Zunjare, P., Chen, W., Tseng, Y., & Vu, T. (2025). SealQA: Raising the Bar for Reasoning in Search-Augmented Language Models. arXiv:2506.01062.