A single ReLU gate fixes distillation's habit of rewarding wrong answers

Standard on-policy distillation (OPD) has a quiet flaw: it pushes a student model to match a teacher's token-level probability distribution without ever asking whether the resulting trajectory is actually correct. The reward signal is purely relational—how much more confident is the teacher than the student at each token—and that ratio can point in the wrong direction. On a correct trajectory where the student is already more confident than the teacher, OPD applies a negative gradient, penalizing a token that contributed to a right answer. On an incorrect trajectory where the teacher outranks the student, OPD applies a positive gradient, reinforcing a token that led somewhere wrong. Both cases violate the basic RL principle that correct trajectories should receive non-negative advantages and incorrect ones non-positive.

OPDVR fixes this with a single ReLU gate. The log-ratio term that OPD uses as an implicit token-level reward is clipped to zero whenever its sign conflicts with the verifier's verdict on the trajectory. For correct trajectories, only tokens where the teacher is more confident than the student pass through; for incorrect trajectories, only tokens where the student is more overconfident than the teacher get penalized. No new hyperparameter is introduced—the gate is a structural correction, not a tunable knob.

The paper's theoretical analysis formalizes what the gate is actually doing: it removes the component of the OPD gradient that opposes the verifier signal, leaving the teacher's distributional guidance intact where it agrees with correctness. Training dynamics bear this out. Across both same-architecture and cross-architecture distillation experiments, roughly 40–50 percent of sampled tokens are zeroed by the gate throughout training—a stable fraction, never collapsing to all or none—confirming that verifier-conflicting tokens are a persistent feature of the data, not an early-training artifact.

The performance gains are real. In the same-architecture setting (Qwen3-4B distilling from a GRPO-trained Qwen3-4B), OPDVR beats standard sampled-token OPD by 2.7 points on AIME24 and 2.1 points on AIME25, and actually exceeds the teacher on AIME24. Cross-architecture gains are larger still: 5.5 points on AMC and 1.7 points on MATH500 over the sampled-token baseline. Combining the gated mechanism with GRPO-style group-relative advantage estimation—the paper calls this GRPD—pushes further, with 6.5 points over GRPO on AIME24 and 10.9 points on AIME25.

The inverse-gate ablation is the cleanest validation: swapping which tokens are kept versus zeroed produces results below vanilla OPD on all six benchmarks, confirming that the specific alignment between gate direction and verifier signal is what matters, not merely the act of masking tokens.