Next-chunk reasoning RL loses to plain mixed SFT once you measure after full RLVR

A persistent assumption in reasoning post-training is that reinforcement learning over unlabeled text—specifically, rewarding a model for predicting the next chunk of a worked solution—must be superior to plain supervised fine-tuning, because it forces the model to reason rather than merely copy. This paper tests that assumption directly and finds it wrong.

The controlled comparison pits two next-chunk reasoning RL variants (next-token reasoning and next-sentence reasoning) against a simple alternative: Mixed SFT, a single fine-tuning stage that trains jointly on no-CoT data and long chain-of-thought data. All five strategies in the study start from the same Qwen3-30B-A3B-Base checkpoint and feed into an identical RLVR stage on DAPO-Math-17K. The verdict is unambiguous. Mixed SFT averages 67.4 post-RLVR across six competition-math benchmarks, roughly 3 points above the next-best RL variant, while consuming over 60% fewer GPU hours. The advantage holds on out-of-domain tasks too—GPQA-Diamond, HLE, and MMLU-Pro—ruling out math-specific overfitting.

The mechanistic analysis is where the paper earns its keep. Next-token reasoning selects training targets by filtering for the top-20% highest-entropy tokens in the no-CoT corpus, on the theory that high entropy marks reasoning-hard positions. The paper shows this filter doesn't work: nearly half of those tokens are already predicted correctly without any reasoning trace, and prediction accuracy climbs substantially during training while entropy stays flat. The model learns to satisfy the reconstruction reward through short, template-like local completions rather than genuine long-horizon inference. Attempting to prevent this collapse by stochastically dropping easy rollout groups and down-weighting positive advantages keeps entropy high but actually lowers the post-RLVR ceiling slightly—entropy collapse is a symptom, not the cause of the ceiling.

Mixed SFT's own behavior is counterintuitive. Its pre-RLVR accuracy of 27.5 sits roughly 20 points below every other method, because jointly training on two structurally incompatible data formats temporarily destabilizes output structure. RLVR's verifiable rewards quickly restore format compliance, and the model then draws on the broader mathematical knowledge absorbed during mixed training, reaching a post-RLVR score of 61.1—a gain of 33.7 points, more than three times larger than any other method's improvement. Sequential SFT, which sees the same data but in two separate stages, ends lower because its second long-CoT stage partially overwrites what the first no-CoT stage deposited, and RLVR cannot recover erased knowledge.

The methodological implication is pointed: evaluating no-CoT training strategies at intermediate checkpoints is misleading. Pre-RLVR accuracy is not a reliable proxy for post-RLVR potential, and any comparison that stops before the full pipeline will systematically misrank its candidates. For practitioners, the practical upshot is that data composition within a single SFT stage beats adding another RL phase.