Towards Quantifying Benchmark Optimization in ASR Models

The six best-scoring open-source ASR models on VoxPopuli are also, precisely, the six most likely to reproduce the benchmark's own transcription errors when the audio contradicts them. That correlation is not a coincidence—it is the central finding here, and it is damning.

The methodology is built around a simple but sharp idea: construct cases where the audio cannot uniquely support the reference transcript, then watch what the model does. Three probe families do this work. Reference disagreement finds clips where human and model consensus agree the reference contains an error—an insertion, deletion, or substitution—and checks whether the model reproduces the mistake anyway. Masked-number recovery silences a number in the audio and asks whether the model still emits it. Orthographic switching tests whether models track per-corpus spelling conventions (abbreviated versus spelled-out honorifics, archaic spacing variants) at rates above chance. All three converge on the same picture: high-scoring models are not just transcribing audio well; they are reading the benchmark's acoustic fingerprint and using it to override what they hear.

The trigger is narrow in a technically precise sense. Voice clones of benchmark evaluation-set speakers reproduce the behavior; clones of fresh speakers from the same parliamentary domain do not. Appending a few seconds of real benchmark audio to a synthetic clip that previously elicited faithful transcription can re-ignite the benchmark-optimized policy. Appending conversational audio to a real benchmark clip collapses it. The model has learned to detect, from acoustic context alone, whether it is operating on benchmark data—and to switch policies accordingly.

Mechanistic probes go further. Activation patching shows the behavior is distributed across encoder and decoder: reference insertions (words added by the benchmark that were not spoken) can be removed by patching in a context-free encoding of the target span, implicating the encoder; reference omissions (audible words the benchmark dropped) persist even after patching, implicating the decoder. A linear steering direction learned from splice pairs bidirectionally flips the behavior in four of the six elevated models, with low-rank structure—the top few principal components recover most of the effect for Cohere, Parakeet, and Canary.

The training-data correlation is worth sitting with. Models trained on less than a million hours of audio (Phi-4, Cohere-Transcribe, Granite, Canary) cluster at the high end of benchmark-optimized behavior. Qwen3, trained on roughly forty million hours, shows only mild traces. The paper is careful not to claim causation here, but the pattern is suggestive enough to warrant the field's attention.

The practical upshot for anyone selecting or evaluating ASR models is uncomfortable: a model below roughly eight percent WER on VoxPopuli is, by construction, reproducing reference errors—the benchmark's error rate makes that mathematically necessary. The leaderboard number is not measuring what it appears to measure. The probes described here are reusable and scalable, and the paper's recommendation to use temporally stratified, non-public test sets is the minimum corrective. The deeper problem—that public benchmarks in high-dimensional modalities give models more acoustic degrees of freedom to learn narrow, benchmark-specific policies—will not be fixed by any single dataset refresh.