VQ tokenizer instability is an optimization design flaw, not a vector quantization limit

VQ tokenizer training keeps breaking not because vector quantization is fundamentally fragile, but because the encoder-decoder and the codebook have always been optimized as a single entangled system when they shouldn't be. StableVQ's central argument is that each module has a distinct responsibility—the encoder should optimize reconstruction under discrete regularization, the codebook should track the encoder output distribution—and that coupling them forces the whole system to depend on accidental cooperation rather than principled individual competence.

Three failure modes motivate the work. When the codebook initializes too small, tokens get assigned to distant codes, the straight-through estimator passes unreliable gradients back to the encoder, commitment loss spikes, and training can hit NaN before utilization ever recovers. When the codebook initializes too large, a few in-range codes get claimed quickly, the training signal for out-of-range codes saturates, and dead codes become permanent. Even a well-initialized codebook can collapse mid-training if the encoder distribution shifts faster than the codebook can follow, triggering a positive-feedback loop.

The three fixes are each targeted at one of these failure modes. Dynamic STE weights each token's gradient contribution by its relative quantization error within the current batch—the best-matched token for a given code gets the full gradient, farther tokens get attenuated proportionally. Crucially, when training is healthy and all tokens are well-assigned, the weights all equal one and the intervention disappears entirely. Region VQ Loss addresses the asymmetry in the standard VQ loss: only selected codes get explicit targets, while inactive codes receive only diffuse indirect signal through the shared projector. Region VQ propagates targets from active codes to nearby inactive ones via a FIFO queue, giving every code a principled local objective. In a frozen-encoder pilot, the standard VQ loss stagnates at roughly 12.5% utilization after 5,000 steps; Region VQ Loss reaches full utilization by step 500. Decoupled Schedule simply assigns the encoder a warmup-plus-annealing schedule and the codebook a constant high learning rate—recognizing that continuous distribution tracking needs sustained gradient magnitude from the first step, not warmup.

None of these components add learnable parameters. StableVQ uses a single linear shared projector by default, yet matches or surpasses methods that require ViT-block projectors with architecture-specific constraints on codebook size. The UR-AUC robustness metric—average codebook usage recovery across different initialization-induced distribution mismatches—shows StableVQ recovering full utilization rapidly across diverse conditions where SimVQ and FVQ recover slowly or not at all. On the non-Gaussian distribution-fitting benchmark, Region VQ maintains 99.4% utilization at the hardest setting, where Wasserstein VQ drops to 34.8% and MMD VQ to 75.6%, while staying close to Wasserstein VQ in per-step training time.

The practical implication is that initialization sensitivity and mid-training collapse—long treated as unavoidable costs of scaling codebooks—are artifacts of entangled optimization, not properties of VQ itself.