Reconstruction FID misleads when choosing image tokenizers for unified models

Reconstruction FID is not a reliable proxy for how well an image tokenizer will behave inside a unified multimodal model. That is the central claim here, and the paper earns it through a controlled experiment rather than assertion.

The setup is a pure-autoregressive continual pretraining testbed built on Qwen3 language models at three sizes, trained on a mixture of roughly 60 million image-text and pure-text samples. Rather than reporting only final benchmark scores, the authors track four task-specific validation losses throughout training: text, unconditional image, text-to-image (T2I), and image-to-text (I2T). The key move is treating these losses as diagnostic instruments rather than as a single aggregate signal.

The first finding is that these losses scale differently and rank tokenizers differently across tasks. UniTok achieves the lowest T2I loss while carrying the highest text loss; GigaTok inverts that pattern. Averaging them obscures exactly the trade-offs that matter. This alone should make practitioners skeptical of any single-number tokenizer leaderboard.

The second finding concerns cross-tokenizer comparability. T2I loss shifts with the image-token vocabulary, so the same numerical loss does not mean the same generation quality across tokenizers. I2T loss, computed over a shared text vocabulary, turns out to be the more stable cross-tokenizer signal—it correlates with generation quality both before and after supervised finetuning, and shows moderate correlation with general VQA performance post-finetuning as well.

The third finding is the most counterintuitive. Swapping GigaTok's standard PatchGAN discriminator for a DINO-based one substantially improves reconstruction FID, yet none of the joint validation losses improve meaningfully, GenAI-Bench score is essentially unchanged, and VQAv2 drops. Semantic supervision on UniTok does the opposite: it worsens reconstruction fidelity but improves all three reported validation losses and both downstream metrics. The mechanism appears to be that semantic supervision tightens the association between image tokens and object-level words—pointwise mutual information between image tokens and COCO object categories rises—rather than simplifying the local token sequence structure.

A fourth finding is less expected: the choice of image tokenizer affects text modeling difficulty even when the text data, text tokenizer, and language backbone are all held fixed. The effect traces specifically to the image-token prediction objective. Ablating T2I training eliminates the text-loss gap between tokenizer pairs; ablating I2T training does not. This cross-modal interference is invisible to any evaluation that looks at generation or understanding in isolation.

Vocabulary size adds a further wrinkle. Among IBQ variants, the relationship between vocabulary size and task-specific loss is non-monotonic—the intermediate size achieves the lowest normalized T2I and image losses—yet the largest vocabulary still wins on downstream benchmarks, plausibly because higher reconstruction fidelity compensates.

The testbed has real limits: seven tokenizers, fixed token length, single-codebook designs only, one backbone family. The authors are explicit that tokenizer rankings may shift under a different language backbone. But the framework itself—track task-specific losses, use I2T loss as a cross-tokenizer diagnostic, check for text-side interference—is transferable and fills a gap that isolated reconstruction benchmarks cannot.