V-RAE: Rethinking Video Latent Spaces for Generation

Most video generation pipelines treat the autoencoder as a solved problem: compress pixels, reconstruct pixels, hand the latent space to a diffusion model, and move on. V-RAE challenges that assumption directly. The core argument is that a latent space optimized for reconstruction fidelity is not necessarily a good space for a generator to learn, and the paper backs this up with a striking empirical finding: rFVD and gFVD produce markedly different rankings of video autoencoders, with Pearson correlations of only 0.200 on UCF101 and 0.473 on K600. A tokenizer that reconstructs beautifully can still impose a latent geometry that a DiT struggles to model.

The proposed fix is to skip reconstruction-driven latent learning entirely and instead use the feature space of a frozen vision foundation model as the generative latent space. A lightweight temporal attention pooling module — roughly 3M parameters — compresses temporally dense encoder features into a compact sequence, and a spatiotemporal Transformer decoder reconstructs pixel-space video from those compressed features. Only the pooler and decoder are trained; the encoder stays frozen throughout.

The pooling design choice matters more than it might appear. Mean pooling preserves semantic structure well but reconstructs poorly. Convolutional pooling improves reconstruction at a steep semantic cost. A much heavier Q-Former design fails to improve either dimension. Temporal attention pooling threads the needle: content-adaptive temporal weighting with a shared 1D query per head, initialized to behave like mean pooling and allowed to depart only when reconstruction evidence warrants it.

The generation results are the most compelling part. Under matched DiT settings with a fixed latent token budget of 1,280 tokens, V-RAE with V-JEPA 2.1 achieves gFVD of 117.86 on UCF101 and 19.16 on K600, outperforming all evaluated VAE-based latent spaces. Convergence is substantially faster — the V-JEPA variant reaches, in roughly 30K updates, a gFVD comparable to what Wan2.2 VAE achieves after 150K updates.

The paper also introduces tFVD, a diagnostic that replaces interior latent codes with temporal midpoints before decoding and measures how coherently the decoder handles those off-trajectory states. The intuition is that a generator must produce latent samples it has never seen encoded, so a latent space whose temporal geometry is smooth and decoder-robust should be easier to model. tFVD correlates with gFVD at 0.919 on K600 and 0.621 on UCF101 — substantially stronger than rFVD in both cases.

The future prediction experiment on Cityscapes adds a useful stress test beyond class-conditional generation. Despite worse reconstruction fidelity than Wan2.2 VAE, V-RAE reduces gFID from 15.02 to 11.52 and gFVD from 144.47 to 111.36 under the same conditional DiT and training budget. The tFVD gap between the two latent spaces on Cityscapes mirrors the generation gap, reinforcing the diagnostic's utility.

The honest limitations: experiments are confined to moderate-scale benchmarks, and the paper does not systematically disentangle how different pretraining objectives or architectures shape the reconstruction-generation trade-off. Whether the advantages hold at the scale of open-domain text-to-video generation remains an open question.