Skill retirement flatlines past a ~45% false-pass rate — and no amount of data brings it back

Notes on The Blind Curator: How a Biased Judge Silently Disables Skill Retirement in Self-Evolving Agents (arXiv:2607.07436) — Xing Zhang, Ya Cui, Guanghui Wang, Ziyuan Li, Wei Qiu, Bing Zhu, Pei-Gen He · July 2026

Note published July 25, 2026 · written by SkillFed’s research pipeline from the paper above · how these notes are made

Self-evolving agents that keep accumulating skills need a curator — a mechanism that retires a skill once its observed pass rate drops to a set threshold, which is what keeps a growing library from drifting below a no-skill baseline. That non-divergence guarantee, built into the Ratchet governance recipe this paper extends, assumes the judge scoring each trial is unbiased. Reference-free tasks like long-form report writing have no ground truth to check answers against, so the only available judge is an LLM judge — and LLM judges run systematically biased, not just noisy. That bias splits into two distinct failure modes, mathematically and behaviorally: symmetric noise, where labels flip in either direction at random, and false-pass bias, where real failures get reported as passes. Both get tested on a 155-slice citation-grounded report-writing testbed — a deterministic five-check grader stands in for ground truth, with corruption injected on top — then cross-checked on MBPP+ code generation, where unit tests play the same role.

The two corruption types behave completely differently. Symmetric noise just compresses the retirement signal — it inflates the effective threshold and the sample size needed, but skills that deserve retirement still eventually get retired. False-pass bias displaces the signal instead: past a rate of (1−τ)/2, about 0.45 under this calibration, no skill is ever retired again, at any sample size — a cliff, not a slope. Genuine contribution-based retirement, isolated from routine cap-eviction churn, collapses to near zero past that threshold across every domain and failure-abundance regime tested, including the code-generation cross-check with a different verifier and composer model entirely. What doesn't generalize is the downstream damage: eval quality only degrades where the same bias also starves skill creation (the scarce-failure subset), and holds steady wherever failures stay abundant enough to keep generating new skills even as the curator dies — so a disabled curator can sit inside a system whose aggregate metrics look completely healthy. A cheap defect-injection audit — planting known defects and checking who catches them — measures a judge's false-pass rate offline, telling an operator which side of the cliff they're on before trusting a system to self-govern.