Frozen video world models already have the controls you need — no fine-tuning required

The central bet here is that frozen video world models already contain everything needed to accept new forms of control — you just have to speak their language. That language is clean visual states read through native self-attention, and World in World (WiW) converts every control signal — depth-projected source frames, rendered human geometry, retrieved historical generations — into exactly that format, without touching a single pretrained weight.

The problem being solved is real and underappreciated. Camera-controlled video rerendering sounds straightforward until you consider what breaks: large viewpoint swings expose surfaces the source video never showed; appearance-based attention drifts when textures repeat or lighting changes; and autoregressive models evict earlier context from their rolling cache, so when the camera revisits a region, the model has forgotten what it generated there. WiW addresses each failure mode with a dedicated evidence source, but all three feed into the same attention pathway.

Two mechanisms do the precision work. Correspondence-guided attention routing (CGAR) uses tracked point identities and camera geometry to steer queries toward geometrically matched source tokens, bypassing the ambiguity that appearance matching alone produces under large viewpoint changes. Evidence-wise attention CFG (EWA) then controls how strongly each evidence channel nudges generation — crucially, it isolates only the component of the evidence response that is complementary to the model's native response, amplifying that while leaving the baseline generation direction alone. Both operate inside the existing denoising forward pass, so EWA adds zero additional network evaluations for guidance.

The ablation results make the architecture legible. Removing target-view warping causes the largest degradation across all reported metrics, with rotation and translation errors rising substantially above full-method values. CGAR and EWA together preserve subject appearance and background structure in ways that neither achieves alone. Rendered body geometry — reconstructed via LHM++ and driven by per-frame SMPL-X parameters — handles the specific failure case of newly exposed human surfaces. Historical retrieval handles the cache-eviction problem for long-horizon revisits.

On DAVIS and OpenVid-1M, WiW achieves the highest average across all seven reported VBench dimensions and the lowest camera trajectory errors among the six compared methods. The applications section extends the same interface to bullet-time, video stabilization, editing, and K/V sharing between two independent generation instances — the last of which is genuinely novel framing: two frozen model instances communicating by passing cached attention features as visual evidence.

The dependency on LHM++ avatar reconstruction for human subjects is a real constraint; non-human or heavily occluded subjects lack that rendered geometry channel. The system also inherits whatever geometric errors DepthCrafter introduces into the depth estimates used for projection. Neither limitation undermines the core contribution, which is architectural: reframing world-model control as an evidence construction and routing problem rather than a fine-tuning problem.