GRPO narrows reasoning diversity while evolution strategies preserve it

GRPO has a well-documented failure mode: it improves single-attempt accuracy while quietly degrading the model's ability to find correct answers under repeated sampling. This paper makes the case that Evolution Strategies (ES) sidestep that failure by design, and it does so with enough theoretical and empirical weight to be taken seriously.

The core argument is about entropy collapse. GRPO's token-level policy gradient tends to concentrate probability mass on a narrower set of reasoning paths, so Pass@K at large K can fall below the base model even as Pass@1 rises. The paper documents this in 15 of 18 comparisons in its Easy Setting experiments across Qwen2.5 and Llama-3.2 models trained on GSM8K. ES, which updates parameters by aggregating reward-weighted perturbations across a population of perturbed models rather than backpropagating through a single policy, avoids this narrowing. Across both Easy and Hard settings, ES improves Pass@1, Pass@16, and Pass@32 simultaneously relative to the base model, while GRPO's gains concentrate at Pass@1.

The theoretical account is built around Jensen-Shannon diversity across the ES population. Because each perturbed model is a distinct policy, sampling across them raises the probability that at least one finds a correct answer. The paper formalizes conditions under which reward weighting amplifies this effect and the resulting center-model update preserves it. The chain is plausible and the proofs are provided, though the gap between the idealized analysis and stochastic autoregressive generation is real.

The forgetting question is handled more carefully than prior work. ES moves 40 to 44 times farther from initialization than GRPO in weight space, a fact previous researchers used to predict catastrophic forgetting. This paper shows that the functional effect of those updates is concentrated in a small subset of larger-magnitude changes—zeroing out coordinates below a magnitude threshold preserves most of the Pass@1 gain until sparsity climbs very high. The largest ES updates cluster in LayerNorm weights and attention projections, while GRPO's largest changes land in token embeddings and the language-model head. Held-out evaluations across five non-training benchmarks show ES preserving or improving performance, not degrading it. The earlier forgetting reports appear to reflect training-set overfitting rather than a general property of ES.

The practical upshot for anyone building reasoning pipelines: the two methods are complementary. Sequential ES→GRPO achieves the highest Pass@32 on the Hard Setting math average while retaining most of GRPO's Pass@1 gain. The required population size also shrinks as model scale grows, which matters for deployment cost. Two-point ZO estimation, popular in supervised fine-tuning, provides no advantage here because regenerated autoregressive responses break the covariance that paired subtraction requires.