Critic-based RL for LLMs was never fundamentally broken — just badly implemented

The central problem BPCO addresses is not that critics are fundamentally weaker than group-based methods like GRPO — it is that standard critic implementations contain several compounding design errors that make instability look inevitable. The paper works through these errors one at a time, using a deliberately small sanity test: fine-tuning a 1.5B model on 1,460 problems it can already solve. Any recipe that fails to reach near-perfect reward on that dataset has an optimization problem, not a capacity problem.

The diagnosis is specific. PPO's ratio-clipping treats low- and high-probability tokens asymmetrically, so switching to DPPO — which clips based on absolute probability shift rather than ratio — is the first fix. A linear value head can predict outside the known reward range, so bounded predictions via a scaled arctangent are the second. Bootstrapped value targets create a self-referential feedback loop where explained variance climbs toward one even as the critic remains inaccurate against actual observed returns; using a pure Monte Carlo target breaks that loop. Batch-wise advantage normalization is perhaps the most insidious problem: as the policy approaches optimality and advantages shrink toward zero, dividing by a near-zero standard deviation amplifies noise into a large spurious training signal and can even flip the sign of positive-advantage examples. Removing normalization lets the update naturally diminish when there is little left to learn. Finally, a fixed GAE lambda weights the terminal reward exponentially less for longer responses, so length-adaptive GAE keeps that weighting roughly constant regardless of response length.

The privileged critic idea is the most architecturally interesting piece. Because the critic is discarded after training, it can receive information the deployed policy never sees — a reference answer, an official solution, a grading rubric. This is the centralized-training / decentralized-execution pattern from multi-agent RL applied to single-model LLM training. The gains are real but conditional: in the small-data sanity test, privileged inputs accelerate critic learning while also accelerating overfitting. On the larger 40.3K-problem dataset, providing the reference answer improves both critic accuracy and held-out AIME performance. On the rubric task, privileged information raises explained variance but does not improve policy performance, suggesting the task was already simple enough that the critic's approximation burden was not the bottleneck.

The scale experiments are credible. BPCO consistently beats a strong critic baseline that already includes decoupled GAE and Monte Carlo targets — the only differences are bounded value predictions and unnormalized advantages. On Qwen3-30B-A3B, the critic baseline stops improving AIME accuracy after roughly 100 training steps; BPCO continues improving throughout. Against the group baseline using 16 responses per prompt, BPCO with one response per prompt matches or exceeds performance across the tested configurations.

The limitations are stated plainly: evidence covers only mathematical and rubric reward signals, the method assumes a known reward range, and the trajectory-matched comparisons do not account for the additional compute and memory that critic training requires. Those are real constraints, not cosmetic ones.