skillfed
$npx skillfedfor your agent

Compiling a skill per task beats a static library by up to 13 Pass@1 points

Notes on Skills on the Fly: Test-Time Adaptive Skill Synthesis for LLM Agents (arXiv:2605.16986) — Jingxing Wang, Chenyue Zhou, Zhihui Fu, Jun Wang, Weiwen Liu, Weinan Zhang, Jianghao Lin · May 2026

Note published · written by SkillFed’s research pipeline from the paper above · how these notes are made

AI-assisted notes · reviewed by SkillFed Skill evolution Bridge: evolution × retrieval

SkillTTA treats skill-building as test-time adaptation: instead of shipping one skill per task family, it keeps a pool of past training trajectories, both successes and failures, tagged with lightweight, answer-free task metadata. At test time it embeds the incoming task and does trajectory retrieval, pulling a small top-k set of the most similar past runs by cosine similarity. A synthesis model then turns the current task plus that retrieved evidence into a one-off skill file, covering when to use it, likely failure modes, and non-prescriptive procedure notes, that a fixed solver conditions on for that single task; no weights change. Benchmarks pit this against a static skill library baseline that synthesizes one skill per trajectory cluster and reuses it as-is, plus a memory-learning baseline that updates through repeated interaction, across spreadsheet editing (SpreadsheetBench), household tasks (ALFWorld), and code generation (BigCodeBench).

Against the static-skill baseline, task-specific synthesis raises SpreadsheetBench Pass@1 from 0.397 to 0.505 and BigCodeBench Pass@1 from 0.517 to 0.651 — roughly 11 and 13 points respectively. On ALFWorld it lands within about 3.5 points of success rate of the memory-learning baseline (0.872 versus 0.907), while finishing successful episodes in fewer steps than any other method tested, including that baseline. Ablations on SpreadsheetBench show why. Synthesizing a skill from retrieved trajectories beats simply prompting with the raw trajectories (0.540 versus 0.422 Pass@1). Retrieval should stay narrow: top-3 beats top-5 and top-9. And restricting retrieval to failed trajectories beats mixing in successes, because failures name the specific, recurring mistakes an evaluator penalizes.

Key numbers

SpreadsheetBench Pass@1, static skill vs. per-task skill0.397 → 0.505
BigCodeBench Pass@1, static skill vs. per-task skill0.517 → 0.651
ALFWorld success-rate gap vs. memory-learning baseline0.872 vs 0.907 (~3.5 pts)
Synthesized skill vs. raw trajectory prompting (Pass@1)0.540 vs 0.422
Retrieval depth where accuracy peakstop-3 best; k=5, k=9 underperform

Skills related to this research

wiki-retrieve wiki-retrieve upgrades vault search from page-level to chunk-level granularity, combining contextual prefixes, BM25 sparse retrieval, and dense reranking to reduce retrieval failures. Setup is opt-in with built-in privacy gates: run fully on-machine with synthetic prefixes, or opt into Anthropic API or claude CLI tiers for better context generation.★ 10,007 laravel-testing This skill guides you through testing Laravel 13 applications with either Pest PHP 4 or PHPUnit 12, covering 24 rules across HTTP endpoints, model factories, database state verification, service mocking, and authentication patterns. It detects your project's testing framework automatically and provides syntax examples for both frameworks side-by-side, so you can write fast, readable tests regardless of which one you use.★ 58 seedance-recipes Seedance-recipes provides production-ready recipe patterns for video content across genres: product, lifestyle, drama, music video, landscape, commercial, animation, and more. Each recipe preserves core creative constraints while inviting customization of subject, camera, lighting, and sound. Use recipes as proven starting shapes, not rigid templates.★ 5,445 Second Brain Query Query your personal wiki to find answers, discover relationships between topics, and synthesize knowledge from your collected sources. The skill searches your index and pages, follows internal links for context, and cites sources using wiki syntax. Save valuable analyses back to your wiki as synthesis pages to compound your knowledge over time.★ 643

Related notes

References

  1. Wang, J., Zhou, C., Fu, Z., Wang, J., Liu, W., Zhang, W., Lin, J. (2026). Skills on the Fly: Test-Time Adaptive Skill Synthesis for LLM Agents. arXiv:2605.16986.
  2. Yao, S. et al. (2022). ReAct: Synergizing Reasoning and Acting in Language Models.
  3. Shinn, N. et al. (2023). Reflexion: Language Agents with Verbal Reinforcement Learning.
  4. Wang, G. et al. (2023). Voyager: An Open-Ended Embodied Agent with Large Language Models.
  5. Zhao, A. et al. (2023). ExpeL: LLM Agents Are Experiential Learners.