Rubric-filtered training lifts a 9B model to 32% accuracy — outcome-only filtering caps out at 18%

Notes on SkillCoach: Self-Evolving Rubrics for Evaluating and Enhancing Agentic Skill-Use (arXiv:2607.01874) — Jiayin Zhu, Kelong Mao, Yudong Guo, Dengbo He, Sulong Xu, Simiu Gu, Yutao Yue · July 2026

Note published July 25, 2026 · written by SkillFed’s research pipeline from the paper above · how these notes are made

SkillCoach splits agentic skill-use into four dimensions scored separately — skill selection, skill following, skill composition, and skill-grounded reflection — pulled from real agent rollouts instead of fixed up front. Each task gets its own self-evolving rubric: a first draft built from the gold skill's steps, then reworked across rounds of judging real trajectories, proposing local patches, and keeping only the ones that clear a validation gate on held-out data. That rubric score runs alongside the task's external pass/fail verifier, never inside it — a trajectory that stumbles into the right answer gets scored differently from one that followed the skill as written. To stress-test selection, agents face a distractor-augmented skill library: gold skills buried among overlapping, superficially similar decoys meant to mimic a crowded enterprise repository.

Evolved rubrics outperform the initial draft as evaluators. Gold-keypoint coverage climbs from 71.56% to 83.70% and usability scores rise from 81.53 to 94.33, while hallucinated grading criteria fall from 2.00% to zero across ten held-out task families. Tested across seven models, from Qwen3.5-4B to Opus-4.7 and GPT-5.5, the rubric exposes gaps that final accuracy alone hides: under distractor-augmented libraries, Opus-4.7 passes 80% of tasks while its skill-following sub-score is only 77.2, and GPT-5.5 passes 76% with a 71.8 following score — the verifier calls both a clean win. Selection reliability tracks distractor similarity more than raw count: GPT-5.5's selection F1 falls from 0.84 under random distractors to 0.59 once fifty decoys sit semantically close to the gold skill. Every model tested has a breaking point — Gemini 3.1 Pro's degradation boundary arrives around 45 distractors, DeepSeek-V4-Flash collapses fully near 6,400, and GPT-5.5 and Opus-4.7 hold up through a 50,000-distractor stress test. Filter the supervised fine-tuning data by the evolved rubric instead of by pass/fail alone, and a 9B model climbs from a 14% baseline to 32% accuracy — versus 18% for outcome-only filtering on the same trajectory pool.