Even a Purpose-Built Reranker Nails the Full Skill Set Only 32% of the Time

Notes on Skill Is Not Document: A Query-Conditional Benchmark and Two-Stage Retriever for LLM Agent Skill Routing (arXiv:2606.03565) — Zifei Wang, Wei Wen, Qian Ji, Ruizhi Qiao, Xing Sun · June 2026

Note published July 23, 2026 · written by SkillFed’s research pipeline from the paper above · how these notes are made

LLM agents that compose multiple skills to solve a task depend on a retriever picking the right combination. That's a different problem from picking the right individual skill. A retriever can score every skill against a query correctly and still return two skills that each look relevant but cannot actually work together for that query — query-conditioned skill compatibility isn't recoverable from per-pair relevance scores. The fix here starts from a byproduct most LLM data-synthesis pipelines throw away: when an LLM decides whether a candidate skill belongs in a query's retrieval set, its rejections already record which skills should not be retrieved together for that query. Reject-as-Resource (R3) keeps those rejections and turns them into training signal instead of discarding them. Built on that idea, R3-Skill is a bilingual Chinese-English benchmark: 10,246 skills (deduplicated from 95,212 raw candidates) across 8 thematic domains, 41,592 accepted queries, and 32,828 of the LLM's own rejected pairings, tagged across an eight-class taxonomy of rejection reasons.

The retriever runs in two stages: a bi-encoder first pass (R3-Embedding) and a cross-encoder reranker (R3-Reranker). Folding the reject signal straight into the bi-encoder's contrastive loss backfires. NDCG@10 falls from 81.60% to 79.94%: a shared skill embedding gets pulled toward one query's positive match and pushed away by another query's rejection at the same time, and the two updates partly cancel (formalized as a gradient-cancellation result). Route the same rejected pairs into the reranker instead — as a graded label between wrong and correct rather than a hard negative — and it works: a cross-encoder scores each query-skill pair on its own, so it isn't subject to that cancellation. The full two-stage pipeline reaches Hit@1 = 75.21%, NDCG@10 = 81.73%, and Set-Compat = 31.88% — the share of queries where the whole correct skill set, not just one right skill, shows up in the results. The reranker's Set-Compat gain is more than double on cross-lingual queries (+6.16 points) versus same-language queries (+2.42 points), and on an external SkillRet benchmark the pipeline outperforms a prior baseline across every metric reported.