Video world models need an authoritative state layer, not better prompts

The core diagnosis here is precise: video world models fail not because their visuals are poor, but because they have no authoritative record of what is actually true in the world. Characters that should be dead keep moving. Entity counts drift. Rules specified by a prompt don't persist across interactions. Programmable World Model treats this as an architectural problem and proposes a clean separation: a lightweight engine owns world state, a video model owns appearance.

The representation choice is where the paper does its most careful thinking. Text prompts are too weak—they don't impose geometric constraints. Full 3D scenes with G-buffers and articulated meshes are too strong—they require the system to explicitly resolve every limb pose and deformation at inference time, which is essentially rebuilding a game engine. The paper argues for state-augmented 3D oriented bounding boxes as a deliberate middle ground: each entity gets a position, extent, and orientation in world coordinates, plus persistent identity, semantic category, and a quantized motion direction. That's enough to tell the renderer where something is and what it's doing without specifying how it moves.

A deterministic compiler projects these boxes into pixel-aligned conditioning maps—identity, semantic, and direction—that feed a ControlNet attached to a frozen pretrained video backbone. The video model fills in everything the boxes leave unspecified: texture, articulation, lighting, secondary motion. The engine, meanwhile, runs executable programs written by a coding agent from natural-language instructions, maintaining health values, faction relations, and event triggers that never appear in any frame but govern what happens next.

The benchmark, CombatStateBench, is deliberately coarse: fifty combat clips evaluated on whether the right number of characters appear alive and whether death events are visually realized. A VLM judge sees only RGB output, no ground-truth boxes. Against two prompt-driven baselines, the method reaches 94% Count Accuracy and 98% State Accuracy. On Count Accuracy, the margin over LingBot-World-V2 is 53.25 percentage points; on State Accuracy, it is 90 percentage points over the same baseline. The baselines communicate state changes by switching text prompts, which is exactly the failure mode the paper is designed to fix.

Long-horizon generation runs chunk-autoregressive, with a geometry-aligned spatial memory that lifts completed frames into world space using estimated depth, then reprojects them into future views. This handles the case where earlier observations fall outside the temporal history window—a real problem when camera angles change substantially over hundreds of frames.

Training data comes from gameplay footage processed by an automatic pipeline: camera poses and depth from ViPE, instance segmentation and tracking from SAM3, per-frame 3D boxes from WildDet3D, all assembled without manual annotation. The sources are Cyberpunk 2077, Forza Horizon 6, and Grand Theft Auto V.

The honest limitation is that the benchmark is narrow—combat scenarios, two metrics, fifty clips. Whether the separation of state and appearance holds up in open-domain settings with more complex interaction graphs remains untested. But the architectural argument is sound, and the gap over prompt-switching baselines is large enough to be meaningful.