A corrected world model can replace real execution in RL and still beat it

The core problem this paper addresses is deceptively simple to state: when you train a research agent with reinforcement learning, generating candidate solutions is cheap because batching lets many trajectories share compute, but executing those solutions is expensive because each one needs its own isolated sandbox with real GPU time. As you scale up the number of trajectories, execution cost grows linearly while generation cost stays nearly flat. The environment becomes the bottleneck, and no amount of clever batching fixes it.

The proposed solution, World Model RL (WMRL), replaces real execution with a language model that predicts what execution would return. Since prediction is just a forward pass, it batches like generation and the bottleneck disappears. The catch, which the paper takes seriously, is that the world model is wrong in two distinct ways: systematic bias (its predictions drift from truth in a consistent direction) and random noise (zero-mean but variance-inflating). Left uncorrected, bias introduces a permanent floor on convergence that no amount of additional training removes—a result the paper proves formally in Theorem 3.

The two corrections are practical and theoretically grounded. Online Debiasing fits a monotone recalibration map over a small stream of real-execution results kept running throughout training—roughly 10% of groups get graded by both the world model and the real environment, providing anchor pairs. Inverse-Variance Denoising then fuses the two gradient streams by weighting each inversely by its variance, which Lemma 12 proves achieves strictly lower variance than either stream alone. Theorem 4 shows that with both corrections active, the permanent bias floor contracts to zero as training proceeds, and the variance term shrinks by a factor that grows as the anchor stream becomes more reliable relative to the noisy world model stream.

The empirical results are striking. WMRL cuts training compute by roughly three to four times compared to standard GRPO on real execution, while actually exceeding its performance on held-out benchmarks—gains of up to 3.1 leaderboard percentile points. The 4B model trained with WMRL surpasses a 48B open-weight agent, and the 9B model surpasses a 120B agent, on tasks neither was trained on. The ablation in Table 3 is particularly clarifying: mixing the two reward streams without any correction actually underperforms real-execution GRPO, confirming that the raw world model signal is harmful without the bias and noise mitigations. Online Debiasing contributes more than Inverse-Variance Denoising individually, which matches the theoretical prediction that bias enters the convergence bound at full magnitude while noise enters damped by step size.

The method also transfers to vision-language-action robot policy training, where Robometer predicts task progress from video frames and sparse real rollout outcomes serve as anchors. The combined WMRL approach adds 3.8 success-rate points over the SFT baseline, while either signal alone adds less than 2 points. The construction requires only that rewards be expensive to collect but predictable from the agent's outputs, and that a thin stream of ground truth remain available—conditions that hold in many post-training settings beyond AutoResearch.