Geometry belongs inside the latent space, not bolted on after generation

The core claim here is that 3D inconsistency in video generation is not primarily a training problem—it is a representation problem. When generators evolve appearance-centric latents, geometric coherence has to be bolted on afterward through camera conditioning, auxiliary losses, or post-training. GAE argues instead that if the latent space itself is derived from a geometry foundation model, coherence becomes native to what the generator learns to produce.

The mechanism is a two-stage codec. A frozen geometry backbone (DA3) produces a four-level feature hierarchy. That hierarchy is poorly suited to generation as-is: despite having over three thousand channels per level, the raw features span only around eleven effective dimensions, with condition numbers that make flow modeling numerically awkward. GAE compresses all four levels into a single spatial latent of 64 or 128 channels, then reconstructs the full hierarchy through the original frozen geometry head—meaning the codec cannot cheat by discarding geometry, because the frozen decoder will catch the loss. A separate learned head decodes RGB from the same bottleneck, so appearance and geometry share one compact state.

Organizing that state for flow matching requires two additional steps beyond reconstruction. Token-wise alignment to a C-RADIO teacher improves transport smoothness and semantic organization, but destroys pairwise spatial structure in the process. A relational objective using DINOv2 pairwise similarities then restores that structure without requiring matching channel dimensions. The paper shows this three-step progression in diagnostics before claiming downstream generation benefits.

The controlled comparison is the strongest part of the work. Seven latent families—pixel VAEs, a semantic representation autoencoder, raw DA3 levels, and the two GAE variants—are each dropped into the same DiT-style flow model, trained on identical data with identical schedules, and evaluated on a shared held-out pool of 64 scenes. Under those conditions, GAE-64 leads on FVD on both RealEstate10K and DL3DV relative to the best competing latent, and camera-trajectory error measured by an independent reconstruction model is roughly halved on RealEstate10K. The geometry gains come from decoding depth and point maps directly from the sampled latent, not from post-hoc estimation on generated RGB—a meaningful distinction that most geometry-aware generation work cannot make.

The limitations worth noting: the qualitative long-rollout model is trained separately at larger scale and is explicitly excluded from the controlled ranking, so the headline metrics and the visually impressive 81-view results come from different checkpoints. The geometry targets used during codec training are pseudo-targets from the frozen backbone rather than ground-truth depth, which bounds reconstruction quality by the backbone's own accuracy. And the approach is tightly coupled to DA3 as the geometry foundation; whether the reparameterization strategy generalizes to other backbones is left open.