A learned rubric and soft logits fix the score-collapse problem in video reward models

Scalar drift is the core problem this work attacks: when a multimodal language model scores a generated video on a 1–5 scale without any explicit anchor, its internal standard wanders. Scores collapse into a narrow high band, leaving downstream reinforcement learning with near-flat gradients, and the same video gets different scores depending on how the prompt is phrased. RewardVerse's answer is to insert a dynamic rubric between the evaluation query and the scorer — a structured intermediate that specifies themes, weights, and scoring tips before the model ever sees the video.

The rubric generator takes only the text query as input, deliberately blind to the candidate video. This prevents the generator from producing lenient criteria for weak videos or harsh ones for strong videos. The scorer then reads logits directly from the five rating tokens rather than generating a free-form number, computing a continuous expected value per theme. Free-form integer generation fails badly in practice: the paper's protocol comparison shows format-failure rates of 46.7% and 53.3% for two integer-output variants, versus zero failures for the soft-logits approach.

Rubric injection alone is not enough. A controlled four-variant study on Qwen2.5-VL-7B shows that adding a rubric expands score standard deviation substantially, but pairing it with natural-language output still leaves two-thirds of comparison pairs tied because the model's linguistic prior anchors it to the high end of the scale. The rubric plus soft-logits combination (V3) is the only configuration that meaningfully spreads predictions across the rating range.

The training algorithm, RGPO, runs in two stages under GRPO. Stage 1 fixes seed rubrics generated offline by Gemini-3.1-Pro through a propose-verify-revise loop and trains only the scorer, establishing human-aligned score margins before any dynamic rubric learning begins. Stage 2 jointly optimizes the rubric generator — rewarded by how well its induced scores separate preferred from non-preferred videos — while continuing to calibrate the scorer against human margin labels. Crucially, the two roles receive asymmetric gradient signals: the generator is updated by policy gradient, while the scorer receives only the margin calibration loss, preventing the model from gaming the reward by inflating score differences.

The data budget is striking: 30 preference pairs per dimension, 480 pairs total across 16 dimensions. On the EvalVerse pointwise benchmark, RewardVerse achieves the highest PLCC in 14 of 16 dimensions, with particularly large margins on Logic and Action. On the external VGRB pairwise benchmark, it outperforms the strongest non-oracle baseline on the seen Visual Quality split and generalizes to the unseen Text Alignment split without any targeted training.

Downstream, fine-tuning Wan-2.2-A14B with GRPO using RewardVerse as the reward improves Imaging Quality from 0.640 to 0.653 while leaving other VBench dimensions largely intact. The same experiment with VideoReward improves Imaging Quality to 0.648 but degrades VBench-Quality and VBench-Text, suggesting reward hacking that RewardVerse avoids.

The paper is candid about what remains open: theme-level scores are not individually supervised, only the aggregated margin is calibrated; the 7B backbone caps what RGPO can recover; and scaling to larger models and training sets is unverified.