Training models to avoid their own bad reasoning beats imitating correct answers

Teaching a model to reason better by showing it how to reason worse turns out to be a surprisingly effective strategy. Negative Self-Distillation (NSD) flips the standard self-improvement playbook: instead of conditioning a teacher on the correct answer and asking the student to imitate it, NSD prompts the model to generate its own "careless reasoner" persona, then trains the student to move its token distribution away from that self-generated flawed teacher.

The motivation is concrete. On-Policy Self-Distillation (OPSD), the dominant alternative, hands the teacher the ground-truth solution. That privileged knowledge produces artificially linear, overconfident reasoning traces. Forcing the student to mimic them suppresses exactly the exploratory, self-corrective behavior that hard problems require. The paper measures this directly: after OPSD training, reflection tokens like "wait" or "let me reconsider" drop to roughly 2 per response on average. After NSD training on the same model, they rise to around 7.5 per response.

The engineering challenge is that you cannot simply penalize every token the flawed teacher favors. Ordinary punctuation and function words appear in both good and bad reasoning; indiscriminate unlearning destroys fluency. NSD addresses this with a gating mechanism that activates a penalty only when the negatively conditioned teacher assigns higher probability to a token than a benign reference model does. The penalty itself is sigmoid-bounded rather than logarithmic, which concentrates the strongest gradient signal on mid-confidence tokens and lets high-confidence structural tokens pass through largely untouched.

The numbers across seven math benchmarks - AIME 2024/25/26, HMMT, AMC, OlympiadBench, and MATH-500 - show average absolute gains of 2.3%, 7.5%, and 6.0% for 1.7B, 4B, and 8B Qwen3 models respectively, all without access to ground-truth labels. The 4B model reaches 35.8% on AIME 2024 and the 8B reaches 39.6%. Efficiency is also favorable: NSD needs only one student rollout per sample versus the multiple rollouts GRPO-style baselines require, and the two frozen forward passes (reference and negative teacher) can run in parallel.

The framework's dependence on the model's own ability to generate meaningful negative conditions is its real constraint. On the 1.7B model, gains are modest precisely because weaker models produce weaker negative contrasts. The authors acknowledge this openly. Still, even the crudest conditioning strategy tested - appending a random Wikipedia passage as noise - delivers competitive results on the 4B model, which suggests the mechanism is robust to imperfect negative signals.