Mixing annotations into RL rollouts only works if you decouple the advantage baseline

The central insight here is simple but easy to miss: every training annotation is already a correct answer in the model's own output format, so why not treat it as one more rollout in the policy optimization group? OraRL does exactly this, but the naive version breaks badly. When you append a high-reward oracle to a GRPO group and normalize advantages across the whole mixed set, the oracle inflates the baseline. Rollouts that genuinely outperform the on-policy average can end up with negative advantages—penalized for being good. The paper calls this advantage inversion and measures it carefully: naive oracle injection flips 22.4% of rollouts that GRPO would have reinforced, and 42.5% of groups contain at least one such inversion. Tracking is worst at 38.7%; segmentation escapes mostly unscathed at 4.5%.

The fix is a decoupled estimator. Policy advantages are computed from on-policy rewards only, so no rollout that beats the on-policy mean can receive a negative advantage. The oracle enters separately: a directional gain scales up above-average rollouts in proportion to how far the oracle sits above the current policy distribution, and a detached oracle advantage provides a direct positive target whose weight decays as the policy closes the gap. Neither term touches the on-policy baseline.

Sign-balanced pruning then cuts the nine-rollout group down to four retained rollouts—always the oracle, one positive on-policy rollout, and two negative ones—before the backward pass. This halves step time relative to GRPO with chain-of-thought: 62.4 seconds versus 135.6. It also beats SFT on accuracy while SFT is faster, and beats GRPO without CoT on both dimensions. The pruning preserves both reinforcing and suppressive signals, which magnitude-only pruning cannot guarantee; removing sign balance costs 1.8 points on temporal grounding in ablation.

The resulting model, Video-ORA-9B, improves over its Qwen3.5-9B backbone across all seven task families and scales consistently from 0.8B to 9B under the same recipe. On VSI-Bench it scores 73.1 against 55.0 for GPT-5 and 55.1 for Gemini-3-Pro. Temporal mIoU rises from 62.5 to 66.0 over the prior best; tracking AO from 73.0 to 78.2. Without chain-of-thought, median end-to-end latency on ten-minute videos drops from 29.0 to 24.3 seconds.

The paper is honest about where the method falls short. Spatial tasks requiring complex reasoning still trail some proprietary models. The ReVSI re-annotation experiment reveals that Video-ORA's VSI-Bench lead does not imply robustness to annotation corrections—it drops 14.9 points under the corrected protocol versus 6.3 for the backbone. Backward relative-direction questions expose a systematic bias: 91.5% accuracy on forward questions, 8.3% on backward ones. These are real limitations, not footnotes.