Every Coin Has Two Sides: On the Dual Nature of Generalization in On-Policy Distillation of Large Language Models

On-policy distillation transfers a teacher's reasoning behavior, not its answers to specific problems. That's the central finding here, and it has real teeth: training on problems the teacher never solves produces nearly identical student performance to training on problems the teacher always solves. Filtering by teacher pass-rate is essentially wasted effort.

The more consequential discovery concerns model origin. When teacher and student share a base checkpoint—one is a further post-trained version of the other—OPD aligns their policies broadly. The student approaches the teacher's accuracy not just on the training domain but across languages, reasoning horizons, and entirely different task domains. Train on English math, improve on Chinese math and long-horizon composed problems. Train on math, improve on code. The top-16 token-distribution overlap between teacher and student rises steadily during same-origin training; for cross-origin pairs it stays flat or declines, even though both settings minimize the same KL objective. Same-origin OPD achieves whole-policy alignment; cross-origin OPD achieves distribution-local fitting.

This distinction inverts some intuitions about teacher quality. A cross-origin teacher that is objectively stronger—higher standalone benchmark scores—can produce a weaker student than a same-origin teacher that is nominally inferior. The 7B experiments show this directly: a cross-origin 14B teacher loses to a same-origin 7B teacher on the student's final performance.

The broad reach of same-origin transfer creates a specific problem for multi-teacher OPD. The standard approach routes each prompt to a domain expert, implicitly assuming that a math teacher's influence stays in math. It doesn't. Because each teacher's policy bleeds across domains, changing the mixture ratio between two teachers shifts the student's capabilities everywhere, not just in the assigned domain. The result is a capability seesaw: more of teacher A means less of teacher B's strengths, regardless of which domain each teacher was assigned to cover. One experiment makes this stark—adding more data from the designated math expert actually degraded math performance, because the science/IF expert happened to be same-origin and was transferring math ability through its own trajectories. The math gains came from the teacher not assigned to math.

The practical implication is that prompt routing is not a capability boundary. Diagnosing why a multi-teacher student underperforms on some domain requires examining every teacher in the pool, not just the one nominally responsible for that domain. The experiments here cover reasoning-oriented models across math, code, science, and instruction-following; whether the same dynamics hold for multimodal or tool-using settings is explicitly left open.