One Feedback Pass Takes Skill-Chain Decomposition From 51% to 68% Accuracy

Notes on Compositional Skill Routing for LLM Agents: Decompose, Retrieve, and Compose (arXiv:2606.18051) — Xueping Gao · June 2026

Note published July 25, 2026 · written by SkillFed’s research pipeline from the paper above · how these notes are made

AI-assisted notes · reviewed by SkillFed Skill retrieval

Compositional skill routing formalizes what happens when a query needs more than one skill: decompose it into atomic sub-tasks, retrieve a skill for each, then compose the results into an executable plan. SkillWeaver, the framework built to do this, has three parts — an LLM decomposer, a bi-encoder retriever indexed with FAISS, and a dependency-aware DAG planner that orders and parallelizes the resulting steps. To measure it, the authors built CompSkillBench: 300 compositional queries (150 easy/2-skill, 100 medium/3-skill, 50 hard/4-to-5-skill) run against 2,209 real skills pulled from the public Model Context Protocol (MCP) server ecosystem and sorted into 24 functional categories.

The headline result flips where most routing work spends its effort: retrieval isn't the weak link, decomposition is. Standard LLM decomposition gets the sub-task count right only 51.0% of the time (decomposition accuracy) and lands the correct skill category at rank 1 (category recall@1) just 34.2% of the time on the full 2,209-skill pool. Feed the retriever's own top candidates back into a second decomposition pass — Skill-Aware Decomposition (SAD) — and decomposition accuracy jumps to 67.7% in a single iteration; get the step count exactly right and category recall at rank 1 climbs to 41.2%. The gains hold even when whole categories or a fifth of the skill pool are withheld from retrieval (+35.6% and +23.2% relative decomposition-accuracy gains, respectively), and the full pipeline needs roughly 1,160 tokens of skill context per query versus ~884K for exposing the entire catalog — a cut of over 99%.

Key numbers

Category recall@1, vanilla decomposition (2,209-skill pool)	34.2%
Decomposition accuracy, before → after one SAD feedback pass	51.0% → 67.7%
Context tokens per query vs. exposing the full catalog	~1,160 vs ~884K (−99.9%)
LLM-Direct baseline: category recall@1 despite 90% decomposition accuracy	21.1%
Benchmark scale	300 queries / 2,209 MCP skills / 24 categories

What this means for skill authors

For skill authors, the implication stings: a skill can be indexed and described exactly right for the task and still never surface — because the upstream decomposer asked for the wrong number of steps before that skill got a chance to compete in retrieval. Tuning a skill's name and description for keyword match addresses retrieval; it does nothing for the decomposition stage that gates it. An ablation confirms this directly: on queries where decomposition already had the right step count, SAD's per-step retrieval gain was statistically indistinguishable from vanilla (p=0.97) — essentially all of the benefit came from fixing granularity, not from better vocabulary alignment.

The LLM-Direct baseline is the sharper warning for anyone building a router by pasting the tool list straight into the system prompt. A much larger proprietary model, given 100 skill names including the correct ones, decomposed the query almost perfectly (90% accuracy) — then picked the right skill category only 21.1% of the time, worse than the far smaller retrieval pipeline. Scale doesn't fix routing; structure does. That's directionally the same bet behind a vetted, categorized skill catalog with dual-tool scanning rather than dumping raw candidates into context. The paper doesn't test that system directly, and its category-level metric stops short of confirming the single correct skill gets picked — worth keeping in mind before stretching the analogy too far.

A caveat that matters: the benchmark measures whether the right category of skill was retrieved, not whether the exact skill was selected or the resulting plan executed successfully — a limitation the authors flag themselves. Even with a perfect step count, category recall@1 plateaus near 40% while recall@10 reaches roughly 80%, a real ranking gap that a reranking pilot only partially closes (+10.3% relative). The 76.7% end-to-end chain-completion figure from their pilot execution study is a promising early signal, not the same claim as the headline recall numbers.

Skills related to this research

idea-generator Idea Generator walks founders through structured discovery to uncover product concepts grounded in what they already know or do. It captures context through targeted questions, synthesizes patterns into candidate directions, scores each against five key axes, and writes a shareable product-idea.md document that feeds into downstream planning skills.★ 168 android-development Master Android app development with modern tools and frameworks. This skill equips you to write production-ready mobile applications using Kotlin's concise syntax and Jetpack Compose's declarative UI toolkit. Streamline your workflow from project setup through deployment on CoWork OS.★ 406 Lesson Study Cycle Designer Lesson Study Cycle Designer structures the Japanese jugyō kenkyū process for teacher teams investigating student learning challenges. It generates a research theme, detailed cycle plan with meeting agendas, a research lesson designed for collaborative observation, and protocols focused on case students. The skill grounds itself in evidence from Stigler & Hiebert, Lewis, Perry & Murata, Dudley, and Takahashi & McDougal on how teachers develop shared professional knowledge through structured cycles of planning, teaching, observing, and analysing.★ 472 Android Development This skill equips you with the core knowledge needed to create functional Android applications. You'll progress through essential concepts, design patterns, and practical coding techniques that form the foundation of professional mobile development. Ideal for developers ready to enter the Android ecosystem.★ 16

Related notes

References

Xueping Gao. "Compositional Skill Routing for LLM Agents: Decompose, Retrieve, and Compose." arXiv:2606.18051 (2026).
Zheng et al. "SkillRouter: Retrieve-and-Rerank Skill Selection for LLM Agents at Scale." arXiv:2603.22455 (2025).
Yuan et al. "CRAFT: Customizing LLMs by Creating and Retrieving from Specialized Toolsets." ICLR (2025).
Karpukhin et al. "Dense Passage Retrieval for Open-Domain Question Answering." EMNLP (2020).
Johnson, Douze, and Jégou. "Billion-Scale Similarity Search with GPUs." IEEE Transactions on Big Data (2019).