Compiling a skill per task beats a static library by up to 13 Pass@1 points
Notes on Skills on the Fly: Test-Time Adaptive Skill Synthesis for LLM Agents (arXiv:2605.16986) — Jingxing Wang, Chenyue Zhou, Zhihui Fu, Jun Wang, Weiwen Liu, Weinan Zhang, Jianghao Lin · May 2026
Note published · written by SkillFed’s research pipeline from the paper above · how these notes are made
AI-assisted notes · reviewed by SkillFed Skill evolution Bridge: evolution × retrievalSkillTTA treats skill-building as test-time adaptation: instead of shipping one skill per task family, it keeps a pool of past training trajectories, both successes and failures, tagged with lightweight, answer-free task metadata. At test time it embeds the incoming task and does trajectory retrieval, pulling a small top-k set of the most similar past runs by cosine similarity. A synthesis model then turns the current task plus that retrieved evidence into a one-off skill file, covering when to use it, likely failure modes, and non-prescriptive procedure notes, that a fixed solver conditions on for that single task; no weights change. Benchmarks pit this against a static skill library baseline that synthesizes one skill per trajectory cluster and reuses it as-is, plus a memory-learning baseline that updates through repeated interaction, across spreadsheet editing (SpreadsheetBench), household tasks (ALFWorld), and code generation (BigCodeBench).
Against the static-skill baseline, task-specific synthesis raises SpreadsheetBench Pass@1 from 0.397 to 0.505 and BigCodeBench Pass@1 from 0.517 to 0.651 — roughly 11 and 13 points respectively. On ALFWorld it lands within about 3.5 points of success rate of the memory-learning baseline (0.872 versus 0.907), while finishing successful episodes in fewer steps than any other method tested, including that baseline. Ablations on SpreadsheetBench show why. Synthesizing a skill from retrieved trajectories beats simply prompting with the raw trajectories (0.540 versus 0.422 Pass@1). Retrieval should stay narrow: top-3 beats top-5 and top-9. And restricting retrieval to failed trajectories beats mixing in successes, because failures name the specific, recurring mistakes an evaluator penalizes.
Key numbers
| SpreadsheetBench Pass@1, static skill vs. per-task skill | 0.397 → 0.505 |
| BigCodeBench Pass@1, static skill vs. per-task skill | 0.517 → 0.651 |
| ALFWorld success-rate gap vs. memory-learning baseline | 0.872 vs 0.907 (~3.5 pts) |
| Synthesized skill vs. raw trajectory prompting (Pass@1) | 0.540 vs 0.422 |
| Retrieval depth where accuracy peaks | top-3 best; k=5, k=9 underperform |
Skills related to this research
Related notes
- Best skill-generation pipeline clears just 14% of code-repo tasks, 25% of document tasks →
- Agents Writing Their Own Skills Close Just 45% of the Gap to Human Authors →
- LLM-Written Agent Skills Add Zero Pass-Rate Gain — Self-Diagnosis Closes Up to 67% of the Gap to Human Authors →
- Diagnose, don't rewrite: three revision rounds take a skill from 36% to 62% success →
- A 3.9M-parameter skill sequencer closes 80% of the gap to hand-picked "gold" skill sets →
- LLM-generated skills move data-science accuracy 1.2 points — same as filler text →
- Executable skills beat prompted skills 51% to 20.5% on web search →
- Four-Way Attribution Gates Every Update to a Million-Scale Skill Corpus →
References
- Wang, J., Zhou, C., Fu, Z., Wang, J., Liu, W., Zhang, W., Lin, J. (2026). Skills on the Fly: Test-Time Adaptive Skill Synthesis for LLM Agents. arXiv:2605.16986.
- Yao, S. et al. (2022). ReAct: Synergizing Reasoning and Acting in Language Models.
- Shinn, N. et al. (2023). Reflexion: Language Agents with Verbal Reinforcement Learning.
- Wang, G. et al. (2023). Voyager: An Open-Ended Embodied Agent with Large Language Models.
- Zhao, A. et al. (2023). ExpeL: LLM Agents Are Experiential Learners.