Synthetic human videos built from robot data may beat manual demos for task transfer

Zero-WAM treats zero-shot robotic task generalization as an in-context learning problem, and its central bet is that a human video is a better task specification than a sentence. Language underspecifies manipulation: it can describe what to do but rarely conveys the intermediate visual states, the spatial constraints, or the temporal rhythm of a task. A short clip of a human hand doing the thing provides all of that directly.

The core technical challenge is that human-robot paired data at the task level is scarce and expensive. The paper's answer is to generate it. An automated pipeline takes task-sampled robot trajectories, runs them through a vision-language model to extract task semantics, edits the first robot frame into a plausible human scene, then synthesizes a human manipulation video with a video generation model. A final VLM pass filters for semantic preservation and physical plausibility. The result is HumanGen: 74.2K human-robot in-context learning pairs spanning 8.6K tasks, assembled without a single manual human demonstration.

The second challenge is subtler. A model trained on seen tasks can learn to ignore the human video entirely, predicting the next robot action from robot history and text alone. This shortcut works fine on familiar tasks and fails silently on novel ones. The paper's countermeasure is the in-context future chunk prediction (IFP) objective: during training, auxiliary modules predict multiple strided future robot video chunks from the current robot-video representation, but without direct access to the human video. The only path for task information from the human video into those predictions runs through the main video branch, which forces that branch to actually encode what the human video is showing.

On seven unseen tasks in RoboTwin 2.0 simulation, Zero-WAM outperforms the strongest video-action baseline, LingBot-VA, by a substantial absolute margin in average success rate. Stack blocks three, the hardest long-horizon task, is the only one where any of the main comparison methods achieves non-zero success, and that method is Zero-WAM. In real-world trials on a bimanual Franka robot, Zero-WAM conditioned on human video beats LingBot-VA conditioned on detailed language descriptions across all three task families, including a precision insertion task where LingBot-VA scores zero.

The ablations are honest about what is doing the work. Task-balanced robotic pre-training alone, without any human video conditioning, already outperforms LingBot-VA by several points, suggesting that the standard practice of sampling trajectories by raw frequency rather than by task is quietly hurting cross-task transfer. IFP adds further gains, especially on articulated-object and long-horizon tasks. The combination is what reaches the reported ceiling.

The paper is candid that experiments focus on stationary tabletop manipulation and that the embodiment gap between human and robot video remains a real constraint. Egocentric human video at internet scale is the obvious next frontier, and the authors flag it explicitly. The architecture is not novel in isolation—it builds directly on the LingBot-VA causal video-action framework—but the data construction pipeline and the IFP objective together address a concrete failure mode that prior work left open.