Wrong pseudo-labels still work because penalizing disagreement is right 79% of the time

The central insight here is statistical rather than architectural: when a majority-vote pseudo-label is wrong on a hard competition problem, roughly 79% of the rollouts that disagree with that label are also wrong. That asymmetry is the entire foundation of TTPO. Penalizing disagreeing rollouts is almost always correct, regardless of whether the pseudo-label itself is. Distilling toward a wrong pseudo-label, by contrast, corrupts the teacher at every token position rather than just once per trajectory. The paper builds a training objective around this gap.

The mechanism splits rollouts into two branches. Agreeing rollouts receive forward-KL distillation toward a teacher conditioned on the pseudo-label answer — and crucially, even when that answer is wrong, the teacher is conditioned on the same answer the student produced, so the update degrades gracefully into thinking-to-non-thinking distillation rather than toward an arbitrary error. Disagreeing rollouts receive GRPO penalties that require only the fact of disagreement, never the pseudo-label's content. Token-level selection sharpens both: the distillation branch down-weights positions where student entropy and teacher-student divergence are both low (already converged), while the RL branch masks out locally correct tokens and focuses penalties on positions where the model was confidently wrong.

The benchmark numbers are striking in one direction that the abstract undersells. Non-thinking evaluation — where the model is tested with thinking mode disabled after training with a thinking-mode teacher — shows gains of +25.2, +30.6, and +36.4 points across the three model scales, compared to +7.1, +5.8, and +3.5 for label-supervised OPSD. The GRPO branch actively suppresses failure modes in the student's own generation mode; pure distillation only passively aligns toward the teacher without penalizing the student's bad habits.

There is a counterintuitive finding about ground-truth labels. Replacing majority-vote pseudo-labels with actual ground-truth answers hurts performance on hard AIME problems. The reason is mechanical: when the model rarely produces a correct answer, ground-truth routing yields almost no positive samples, starving the distillation branch and leaving GRPO with near-zero advantages. Majority-vote labels, being easier to match, keep both branches active and maintain a healthy positive-negative split. The paper tracks this through entropy curves and loss dynamics, and the training signal comparison is unambiguous.

The self-evolution dynamic is also worth noting. As the model improves, Maj@12 rises alongside Avg@12 — better rollouts produce more accurate pseudo-labels, which raise the training ceiling for subsequent steps. This feedback loop allows TTPO to eventually exceed what training with ground-truth labels achieves, not because the pseudo-labels are better, but because they keep the optimization well-conditioned throughout.

The stated limitations are honest: the method degrades when the sample budget is too small for reliable majority voting, experiments are restricted to mathematical reasoning with extractable final answers, and the fixed asymmetric objective does not adapt as pseudo-label quality improves over training. All three are real constraints on deployment scope.