Cross-shot memory and rollout-aware training finally let video models remember

Video generation has been stuck in a fundamental rut: models produce excellent individual clips but forget everything the moment a new shot begins. JoyAI-Echo-1.5 attacks this directly with two purpose-built variants that share a common training philosophy — make the model remember, make it respond to controls, and make it stable when conditioned on its own outputs.

The long-video variant builds what the paper calls composable cross-shot memory. Rather than relying on a single adjacent frame for character continuity, it aggregates visual evidence from multiple prior shots and derives speaker identity from speech-filtered full-shot audio. The key insight is that a short audio window selected by acoustic energy tends to get dominated by background music, which then gets reinforced during autoregressive rollout. Using the complete filtered speech content of each shot is cleaner and more stable. Memory and target tokens are kept in separated positional regions via RoPE offsets, so the model can simultaneously accept memory, a first-frame image condition, or both without treating them as competing inputs.

The world-model variant solves a different problem: heterogeneous action spaces. Keyboard inputs, game engine logs, and reconstructed camera trajectories from web video all speak different languages. The system converts everything into calibrated metric 6-DoF camera trajectories and injects them through a geometry-aware conditioning pathway, making the model controller-agnostic. On WBench Navigation, the undistilled model scores 81.7 overall, ranking first, while the distilled four-step causal variant scores 81.0 — a gap small enough to suggest that the distillation process preserves most of the interactive capability.

The training pipeline for streaming generation is where the engineering gets genuinely interesting. Converting a bidirectional diffusion backbone into a causal autoregressive generator starts with teacher forcing on clean ground-truth history, then applies Self-Gradient Forcing on self-generated rollouts. The short-horizon version trains robustness to accumulated errors within a single training window. The long-horizon extension constructs trajectories spanning multiple segments, using a sink-plus-FIFO KV cache to keep memory bounded. Crucially, the gradient reach extends beyond the direct attention window through multi-hop paths across Transformer layers — a loss on a late chunk can supervise context-writing operations in much earlier chunks without retaining the full sequential autograd graph.

The super-resolution stage adds an unexpected contribution: rather than committing to a fixed step count at distillation time, it uses MeanFlow to train a model of average velocity over an interval. This makes the step count a runtime dial — one set of weights serves any budget from a single step upward, without retraining.

On the long-video benchmark of 100 stories and 3,000 shots, the system leads on six of seven metrics, with the largest improvements in speaker consistency and cross-shot identity. Human evaluators preferred it over a competing system by wide margins on audio-visual synchronization and instruction following. The paper is candid about what remains hard: accumulated rotational drift on difficult long-horizon trajectories is named explicitly as the key unsolved limitation.