Holistic similarity scores hide the failures that matter most in video generation

Most video generation benchmarks treat reference fidelity as a single number: did the output look like the input? OmniVBench argues that question is too coarse to be useful, and the experimental results back that up.

The benchmark organizes reference-to-video evaluation into 7 task families and 18 fine-grained tasks spanning content, motion, style, structure, narrative, and multi-reference compositions. Its 813 evaluation cases are each decomposed into case-specific checklists—12,172 atomic items in total—that ask not just whether a reference was preserved, but whether the right factor was preserved, correctly disentangled from irrelevant information, and routed to the correct target. That last distinction matters most in multi-reference settings, where a model might faithfully follow a motion reference while accidentally copying the appearance of the subject in that reference onto the wrong character. Holistic similarity scores cannot catch that failure; the checklist protocol can.

The evaluation runs three dimensions in parallel: Reference Fidelity, Instruction Realization, and Video Quality. Instruction Realization is itself split into disentanglement-and-routing versus target compliance, and the results on that split are telling. Several models score reasonably on target compliance while scoring substantially lower on disentanglement, particularly on multi-content and cross-aspect tasks. A model can follow the instruction's stated goal while still bleeding irrelevant reference content into the output—a failure mode that only becomes visible when you measure the two separately.

The companion Omni-R2V Dataset addresses the training-data side of the same problem. At roughly 340K processed samples drawn primarily from professional video footage, it covers the same seven task families as the benchmark. Construction pipelines differ by task: action references are built by animating retrieved character images with motion extracted from target videos; camera-motion references come from three sources including synthetic Unreal Engine footage; style references are cross-matched to minimize semantic overlap while preserving visual style. The dataset provides ready-to-use reference–target pairs rather than raw source material, which is a practical distinction given how much preprocessing effort existing datasets require.

The evaluation of eleven models—six closed-source, five open-source—shows the gap between open and closed systems has narrowed considerably, with the strongest open-source models reaching comparable overall scores to leading closed-source ones. But no model dominates across all task families. Content reference is where scores are highest and most consistent; motion, structure, narrative, and multi-reference settings show the widest spread. Strong content fidelity does not transfer to motion or narrative fidelity, which suggests these capabilities are genuinely distinct rather than reflections of a single underlying competence.

The automatic evaluation is validated against human judgments on 965 model outputs across all 18 sub-tasks. The factor-grounded checklist approach correlates more strongly with human ratings than holistic evaluation does, which is the key methodological claim and the one the ablation in Table 6 directly supports.