PPO's value critic has been broken by dense supervision, and 3 sparse anchors fix it

Standard PPO for LLM reasoning has a quiet critic problem. The value network is supposed to track how a model's chances of success shift across the tokens of a response — but it doesn't. This paper names and dissects that failure: the critic's predictions stay nearly flat even when Monte Carlo estimates of the true state value swing sharply between adjacent reasoning steps. The critic can even move in the wrong direction.

The diagnosis is precise. Two mechanisms drive the flattening. First, when rewards are terminal-only, every token position in a response gets the same regression target — the final outcome. Minimizing mean squared error across all those positions implicitly penalizes variance in the predictions, pushing them toward a uniform value within each response. Second, adjacent LLM states differ by exactly one token, so their hidden representations and gradients are nearly identical. Dense token-level supervision therefore piles up redundant, aligned updates that further homogenize the value profile. The paper formalizes both effects mathematically and confirms them empirically: gradient cosine similarity between positions drops as token distance increases, and the critic's hidden-state trajectory for a single response occupies a strikingly compact region.

The fix is deliberately minimal. SParse PPO (S3PO) applies the critic loss at only a handful of well-separated positions per response — by default three anchors at fixed relative positions plus a tail anchor for long outputs. The actor objective, rollout procedure, and return targets are untouched. Only the set of states receiving the value loss changes.

The results on Qwen3-4B-Base and Qwen3-8B-Base are clear. Against standard PPO, S3PO gains nearly 8 percentage points on in-domain mathematical reasoning and over 7 on out-of-distribution tasks for the 4B model, with consistent improvements at 8B as well. The ablations are informative: performance peaks around three supervised anchors and degrades back toward the dense PPO baseline as anchor count climbs toward full token-level supervision. Random anchor placement underperforms fixed spacing, consistent with the temporal-correlation story. Dropping the tail anchor hurts both accuracy and generation stability.

The FrozenLake control experiment is a useful sanity check. In a setting where ground-truth state values are known, critic predictions become progressively smoother as the maze grows — the same pathology, in a fully observable environment, scaling with state-space size.

One thing the paper is honest about: Monte Carlo value estimates used as diagnostics require multiple independent continuations per intermediate state, which is expensive. S3PO itself does not require them at training time — they are only used for evaluation. The method's actual training overhead over standard PPO is minimal.

The broader implication is that critic-based PPO has been quietly degraded by a supervision structure nobody questioned. The fix costs almost nothing.