Keeping one skill per family drops harmful-sibling exposure from 69% to zero

Notes on SkillResolve-Bench: Measuring and Resolving Same-Capability Ambiguity in Agent Skill Retrieval (arXiv:2606.10388) — Jiandong Ding · June 2026

Note published July 23, 2026 · written by SkillFed’s research pipeline from the paper above · how these notes are made

A retriever can nail the capability family — the right general kind of skill for the job — and still hand back the wrong specific one. SkillResolve-Bench 1.0 names this failure same-capability execution-risk retrieval: each query pairs a helpful skill against a query-specific risky sibling — same capability, but wired to a stale resource, an unmet precondition, or the wrong procedure. The benchmark draws 661 helpful/risky pairs (630 from SRA-Bench, 31 from SkillsBench) and tests them against a 7,982-candidate pool that includes 6,660 public SkillRet skills, split query-disjoint at 446/68/147. A new metric, harmful sibling rate (HSR@K), tracks whether the risky sibling shows up in the top-K results at all.

Relevance-only retrieval fails badly on this benchmark: the SkillRouter baseline reaches a respectable Recall@3 of 0.654 while surfacing the risky sibling in the top 3 results 69.3% of the time. SkillResolve, the paper's reference method, works in three moves. A capability resolver groups competing candidates into families. A utility scorer, trained on confusable library negatives plus contract-profile cues — resource bindings, preconditions, API scope, procedures — ranks what's left. A representative selector then keeps only the single highest-utility member of each family before the final ranking. That combination lifts Recall@3 to 0.766 and NDCG@3 to 0.699 while dropping HSR@3 to zero. Strip out only the representative-selection step and keep the same scorer: ranking quality barely moves (Recall@3 0.762) but the harmful sibling rate jumps back to 0.236 — evidence that picking one representative per family, not better scoring alone, is what suppresses exposure.