HarnessEval-W: Agentifying the Evaluation of Visual Worlds

Most world model benchmarks hand you a number and nothing else. You cannot tell whether a model failed because it rendered the wrong object, ignored the physics of a collision, or simply drifted in style over a long rollout. HarnessEval-W treats that opacity as the core problem and builds an evaluation pipeline that produces a full reasoning trace for every verdict it issues.

The mechanism is hierarchical. A top-level planner reads the context of each test case—initial image, action prompt, evaluation setting—and routes it to skills from a library rather than applying a fixed rubric. Each skill then decomposes its question into sub-questions, spawns dedicated sub-agents to answer them, and a parent agent validates the collected evidence before aggregating a score. The result is what the paper calls an evidence tree: you can trace any score back to the exact sub-question that failed, the tool that supplied the visual grounding, and the logical chain connecting them.

The benchmark covers 330 cases organized around three axes derived from a formal factorization of what a world model must do: render observations from the current state, update that state under actions, and maintain coherence over time. Those axes decompose into eight metrics—render quality, physical observation quality, three transition types, and three persistence settings. Cases were authored by an agentic construction pipeline that samples from a six-axis scene taxonomy, grounds actions in generated images, and runs a validation agent to reject candidates where the target is invisible or the expected outcome is too vague to measure.

The evaluator was validated against 5,000 human A/B pairwise judgments across nine models. Against the closest comparable protocol from WBench—run on the same videos with the same GPT-5.5 backend—HarnessEval-W substantially raises pairwise accuracy and cuts the draw rate on both Physical and Intentional Transition, while also achieving a lower Brier score in both settings. Its evaluation is also more stable across repeated runs: the envelope of three independent runs spans far fewer Bradley–Terry units than WBench's equivalent, whose slope nearly doubles across rounds.

The capability analysis across 18 models is where the benchmark earns its keep beyond leaderboard positioning. Fine-tuning a text-to-video generator into an action-conditioned world model consistently trades physical and intentional transition accuracy for better revisit consistency—the model gets better at returning to places it has seen and worse at executing commanded interventions. That tradeoff is invisible to any benchmark that collapses everything into a single score.

The design has real costs. Every evaluation case runs a multi-agent pipeline with a VLM backend, which is expensive at scale. The skill library is finite and the paper is candid that gaps will appear as models improve—though it frames those gaps as inputs to a self-improvement loop rather than failures. Whether that loop closes fast enough to keep pace with frontier models is an open question the paper does not resolve.