Letting executable code own world causality is the right split for video generation

Video world models have a structural blind spot: they learn from the visible outcomes of world rules without ever seeing the rules themselves. Every frame of a gameplay video is the rendered output of executing game code, but that code is discarded after rendering. The model must reverse-engineer causality from its shadows.

Code World Model attacks this directly by splitting the problem in two. A coding agent maintains an explicit, executable world state - entity positions, faction relations, cooldown timers, event histories - and updates it through code it can write, revise, and reuse. A video diffusion model then renders that state into visual observations. The two components operate at different frequencies by design: code runs high-frequency deterministic updates continuously, while the coding agent intervenes only when a new situation demands reasoning or a mechanism needs changing. This decoupling is the core architectural claim.

The bridge between them is what the paper calls a proxy: a coarse programmatic representation of camera motion, entity positions, poses, and spatial relations, compiled deterministically into a low-resolution video that conditions the visual model frame by frame. The proxy runs at one quarter of the output resolution along each spatial dimension, so it adds roughly as many visual tokens as the target video - negligible overhead. It deliberately omits textures, materials, and fine motion, leaving those to the video model's learned priors. The coding agent controls what happens; the video model controls how it looks.

The prototype fine-tunes MiniMax-H3 with rank-128 LoRA across all 50 transformer blocks, yielding roughly 596 million trainable parameters, on about 5.6 hours of GTA V gameplay compiled into 9,420 five-second clips. Training runs for three epochs on eight H800 GPUs. The paper is candid that this is a small-compute proof of concept: generation quality is limited, real-time autoregressive generation is not implemented, and the coding agent - GPT-5.6 Sol in their setup - still struggles to build highly complex game mechanisms from scratch.

The proxy-construction pipeline extends to real-world video via a KITTI-360 demonstration: calibrated 3D reconstructions and object annotations compile aligned proxy conditions offline, without action labels. This matters because action annotation in unconstrained video is ambiguous; the proxy encodes motion consequences directly rather than requiring a separate label.

The honest limitation is that qualitative results come from simple worlds the coding agent built using existing AAA game scenes as templates. The paper does not demonstrate an autonomously constructed open-world game, and it acknowledges that reliably building complex interactive worlds from scratch remains future work. What it does show is that the proxy interface is learnable and that the agent-code-video decomposition is coherent enough to produce spatiotemporally consistent output from a small training set.