Video models still can't track hidden objects, but synthetic physics data helps
on: Training Object Permanence in World Models
Video generation models fail at something human infants master by three and a half months: knowing that an object behind an occluder still exists. WROP (World Reasoning with Object Permanence) is a systematic attempt to measure and fix that failure.
The benchmark consists of 150 hand-designed Blender generators organized into six task families — three probing object permanence, three probing object solidity. Each generator randomizes surface-level parameters like lighting and camera angle while holding the underlying physical scenario fixed, producing over 10,000 samples per task and a total training corpus of 1.5 million clips. A fixed 300-question exam, two questions per generator, drives the evaluation.
Fourteen models were tested across three interface classes: true continuation, reference-to-video, and edit/transfer. The primary measure is a blind pairwise human preference study with 20 raters, scored via Bradley-Terry on an Elo scale. Reference-to-video models dominate: Wan 3.0 Prime and MiniMax H3 tie at Elo 1724, with Seedance 2.5 fourth at 1650. Their advantage likely comes from regenerating scenes on their own timeline rather than extending a clip frame-by-frame — more freedom, but at the cost of sometimes ignoring what the input established.
PWM-WROP, a 16B model fine-tuned from Cosmos3-Nano on the WROP corpus for one epoch, ranks third overall at Elo 1680 and first among true-continuation models, leading the next-best continuation system by 224 Elo points. It achieves this at a native output resolution of 320×192, while competing systems output at 720p or 1080p; at matched resolution it posts the best LPIPS and MS-SSIM against the target video.
The qualitative breakdowns are the most instructive part. Two failure modes recur across models: representation dropout, where objects fail to re-emerge correctly after occlusion, and causal decoupling, where physically linked events — support removal and falling, collision and momentum transfer — are generated independently rather than causally connected. PWM-WROP handles occlusion tracking well but is less consistent on contact-based dynamics, ranking eighth on the collision family.
The paper is candid about what the result does and doesn't show. Because models differ in architecture as well as training data, the performance gap can't be attributed to fine-tuning alone. The benchmark itself uses human raters rather than vision-language model judges, a deliberate choice given documented VLM deficits on exactly the core-knowledge tasks being tested.
All data, exam questions, model outputs, scores, weights, and the native-PyTorch training stack for AWS Trainium2 are released. The training stack reduced step time from 15.1 seconds to 5.7 seconds through targeted engineering, with the full optimization record documented.
A cognitively grounded benchmark that exposes two distinct failure modes in video models and shows fine-tuning on synthetic physics data meaningfully closes one of them.