Self-distillation fails to recover pruned speech encoders; a larger teacher is essential

Pruning layers from a speech encoder sounds straightforward until the decoder starts hallucinating end-of-sequence tokens after just a few words. That failure mode is the real problem X-AuT is solving. When Qwen3-ASR's audio tower loses complete Transformer blocks, the conditioning embeddings fed into the frozen language-model decoder shift enough that the decoder loses its acoustic footing and collapses outputs. The paper's contribution is a recovery recipe that addresses this without touching the decoder's pretrained weights beyond LoRA adapters on its attention projections.

The framework works in two pruning hops: 18 layers down to 16, then 16 down to 14. Before each hop, short behavioral probes evaluate candidate layer combinations under a matched training budget rather than relying on static importance scores. This matters because layer interactions are non-additive in ways that individual scores cannot predict. The two strongest single-layer removals, when pruned together, produce a pair that underperforms a less obvious adjacent pair by nearly a full percentage point after matched recovery.

Recovery runs in three stages. Stage 0 aligns intermediate and bridge representations using mean-squared error and cosine distance losses, with a learned two-layer bottleneck projecting the 1.7B teacher's 2048-dimensional hidden states into the student's 1024-dimensional space. Stage 1 switches to logit distillation and introduces scheduled student-policy batches: roughly every fifth optimizer step, the student generates a prefix and both teacher and student are evaluated on that generated context. Stage 2 drops the teacher entirely and finetunes with gold-transcript cross-entropy, reweighting sources toward target-domain data.

The teacher-scale comparison is the sharpest result in the paper. Under an otherwise identical recipe, replacing the 1.7B cross-scale teacher with the unpruned 0.6B model produces 8.45% macro error versus 5.55%—worse on all ten benchmarks, and 50% above the unpruned baseline rather than below it. Self-distillation here is not a degraded fallback; it actively fails to recover the pruned model.

The headline numbers: the 16-layer model improves macro-average error from 5.61% to 5.27% across ten Chinese–English benchmarks. The 14-layer model reaches 5.75% with 20.7% fewer audio-tower parameters (147.8M versus 186.4M). On an in-vehicle accelerator, encoder latency drops 21.4%, though end-to-end latency falls only 4.7% because autoregressive decoding dominates.

The paper is candid about what these numbers do and do not establish. All main results are single runs with seed 42. The 0.14-percentage-point gap between the 14-layer model and the baseline is described as a descriptive observation, not a statistically significant finding. The study covers one model family, and the authors explicitly flag that layer interactions, EOS behavior, and alignment dynamics may differ in Whisper or Qwen2-Audio architectures.