Zing-0.5: Toward Playable Worlds with Real-Time Joint Action and Text Control

Zing-0.5 is a 5-billion-parameter autoregressive world model that lets a user steer through a generated environment with keyboard inputs while issuing text instructions mid-session—without restarting generation. The sled demo in the paper is the clearest illustration: a rider navigates a snowy landscape, and the user types instructions to make the rider cheer and open an umbrella, all within the same continuous video stream. That combination—directional control and semantic intervention sharing one visual history—is the actual design target, not a feature added on top.

The architecture sits on Wan2.2-TI2V-5B and adds a lightweight action encoder (roughly 3.68 million parameters, about 0.074% of the backbone) that maps magnitude-aware keyboard channels to frame-aligned features. The eight directional channels—W/A/S/D for movement, I/J/K/L for view—each carry a continuous strength rather than a binary press, which preserves fine-grained motion information that discrete labels would discard. Text prompts are aligned to temporal intervals rather than the whole video, so a prompt update refreshes only the text key-value cache while the visual cache stays intact. That asymmetric cache update is what makes mid-session instruction changes cheap.

The training pipeline is four stages. Bidirectional adaptation introduces action conditioning without destroying the pretrained image and video generation capabilities. Autoregressive adaptation trains two branches: a segment-level teacher that can see an entire prompt interval jointly, and a block-level generator that produces four latent frames at a time. The teacher exists solely to supervise the student through distribution-matching distillation—it never runs at inference. ODE initialization and local consistency distillation follow, and then the student trains on its own rollouts using a rollout-and-replay procedure adapted from Self Gradient Forcing, which separates per-instance generation from packed gradient computation to keep memory tractable on long, variable-length sequences.

The result runs at 480p and 24 FPS on a single RTX 5090, with eight independent streams per eight-GPU server. The estimated server rental cost is approximately $0.009 per stream-minute. Each GPU holds a complete model replica, avoiding cross-GPU latent transfer. A bounded causal KV cache with a fixed prefix sink and a sliding recent-context window keeps memory from growing with session length.

On the 158-case WBench Navigation benchmark, Zing-0.5 scores 81.0 overall and 88.5 for consistency. The paper is candid that these numbers cover navigation-conditioned generation; the text-control capability is evaluated only qualitatively through recorded sessions.

The most honest section of the paper is the discussion of what the model cannot do. Zing-0.5 has no explicit state representation. An object moved aside by a text instruction may not remain displaced when the user returns to it. The bounded KV cache discards older visual evidence, and autoregressive error accumulation means a plausible-looking continuation can silently contradict an earlier event. The paper frames persistent consequences—not just immediate visual responses—as the open problem that future architectures need to treat as a first-class design objective. That framing is correct, and the honesty about it is more useful than the benchmark score.