Cutting 83% of skill calls raises task success 10.9 points

Notes on Skill or Skip? Learning Selective Skill Invocation in Agentic Tasks via Dual-Granularity Preference Learning (arXiv:2606.00510) — Chishui Chen, Jiaye Lin, Te Sun, Junxi Wang, Y. Yang, Congming Qin, Yangen Hu, Lu Pan, Ke Zeng · May 2026

Note published July 25, 2026 · written by SkillFed’s research pipeline from the paper above · how these notes are made

SelSkill treats every point where an agent could call an agent skill as a binary decision — invoke or skip — instead of assuming relevance implies use. It flags candidate decision points with predictive entropy (how uncertain the model is about its next token), then forks the trajectory at that exact prefix into two rollouts: one calls the skill, one doesn't. Both run to completion and get labeled by outcome and step-efficiency — success beats failure, and between two successes the shorter one wins. Those local invoke/skip pairs get combined with whole-trajectory, episode-level preferences (successful runs vs. failed ones) and trained jointly with Direct Preference Optimization (DPO). The loss is masked to just the few assistant turns right after each branch, so the gradient targets the invocation decision itself, not the whole trajectory.

A counterfactual audit across 513 paired runs on BFCL, ALFWorld, and Tau-bench explains why: enabling skill access helps in just 13.5% of cases, changes nothing in 78.2%, and actively hurts in 8.4% — and the harmful calls look just as on-topic as the helpful ones by BM25 and embedding similarity, so relevance-based filtering can't tell them apart. Training on that dual-granularity signal fixes the downstream behavior. On ALFWorld with Qwen3-8B, SelSkill lifts task success by 10.9 points and execution precision by 29.1 points, while cutting invocations per episode from 2.55 to 0.44. On BFCL it adds 5.7 points of task success and 29.5 points of execution precision. The learned invoke/skip judgment transfers zero-shot to Tau-bench and PopQA on entirely unseen skills — on PopQA it keeps retrieval calls for obscure entities but drops the invocation rate on well-known ones from 78% to 57%, with no accuracy cost.