Multi-teacher distillation fails from budget misallocation, not gradient conflict

When you combine three domain-expert teachers into one student via on-policy distillation, the obvious suspect for failure is gradient conflict—the teachers disagreeing on shared tokens like connectives and formatting words. Open-MOPD tests this hypothesis directly and rules it out. Average teacher disagreement sits at 0.126 nats throughout training, well below the 1-nat threshold where teacher preferences meaningfully diverge. Only 0.62% of tokens show strong conflict. Masking or averaging those tokens actually hurts performance. The real problem is elsewhere.

The actual bottleneck is a severe misallocation of the token-level optimization budget. Instruction-following responses average around 409 tokens; math and code responses average roughly 10,500. Because the distillation loss aggregates across tokens, IF receives about 1% of gradient tokens despite accounting for roughly 20% of sampled prompts. The ratio is 25-to-1 in response length, which means IF would need its prompt count scaled up 33-fold to reach parity through oversampling alone—which would crowd out the long-chain math and code supervision entirely.

Length disparity is only the first problem. Even after token shares are equalized, reward magnitudes drift at different rates as the student converges toward each teacher at different speeds. Within 25 steps of equalizing token shares, IF's budget share collapses from 48.7% to around 9% while code's climbs to 63.8%. A third issue compounds both: standard multi-step rollout reuse leaves the student-dependent reward component stale across inner updates, with KL between rollout and current policy growing monotonically—reaching 0.216 at the fourth inner update and clipping 86% of tokens.

Open-MOPD addresses each failure with a dedicated mechanism. Token-share balancing assigns per-domain loss weights that enforce equal gradient-token shares regardless of response length—amplifying each IF token roughly 48-fold. Gap-following allocation then dynamically steers the budget toward whichever domain has the largest remaining teacher-student gap, preventing the feedback loop where a converging domain absorbs ever more budget. The paper demonstrates this loop concretely: reversing the sign of the gap factor causes training to collapse at step 74. Reward refresh recomputes the student-dependent reward term before each inner update using the current student parameters, reusing the actor forward PPO already requires—adding no measurable runtime overhead.

Cumulatively, these three mechanisms reduce the integration gap from 3.50 points to 0.31 points on the six-benchmark evaluation suite. The entire pipeline—mixed-domain SFT, three domain RL teachers, multi-teacher distillation, and all ablations—runs on a single 8×A100-80GB node, a deliberate constraint that makes the recipe reproducible for academic teams rather than just industrial labs.