LEGO-RL: Harness-Native Reinforcement Learning for Coding Agents

The central problem Lego-RL solves is not model capability but infrastructure fidelity. When you train a coding agent with policy-gradient methods, the harness managing tool calls, context compaction, and repository state will silently corrupt your training signal unless you intercept the model at the right boundary. Lego-RL intercepts it there: an in-process proxy sits at the provider API, capturing token IDs, log-probabilities, response masks, and—critically for sparse mixture-of-experts models—expert routing decisions at generation time, before any harness-side rewriting can invalidate them.

The routing-replay finding alone is worth attention. For the sparse Qwen3.5-35B-A3B model used throughout, replaying rollout-time routing decisions during training substantially raises rollout–training correlation and cuts mean token log-probability discrepancy. A one-position offset bug in an intermediate implementation—routing every token through its neighbor's experts—degraded all alignment metrics below the no-replay baseline while the system appeared to function normally. The only diagnostic that caught it was direct comparison of replayed expert assignments against the model's live selections.

The reward-integrity work is equally concrete. Six systematic failure modes were catalogued: agents that expose grading information through network access, environments that produce rewards independent of agent behavior, verifier logic that incorrectly applies the reference patch. A non-trivial fraction of tasks in the initial OpenSWE-derived pool contained verifier logic of that last kind. The defenses—network restrictions via a privileged sidecar the agent cannot modify, test dependencies packaged into the task image, repository history withheld until verification—are enforced inside the sandbox rather than trusted to the harness.

On SWE-bench Verified, training Qwen3.5-35B-A3B with GSPO across three harnesses yields gains of several percentage points over each harness's own baseline, and the trained model exceeds the next base-model generation by more than the base generation itself is worth. The harness-dependence finding is pointed: a model post-trained under Claude Code loses ground under OpenHands SDK relative to the untuned newer base. A gain obtained under one agent control flow need not survive another.

The behavioral analysis is unusually granular. The proportion of trajectories that reread a file after modifying it rises substantially over training, while recovery after intermediate command failures barely moves—suggesting the policy learns verification behavior more readily than error recovery. Response length growth is driven by more turns rather than longer individual turns. Task-level solve rates are heterogeneous: roughly half of tasks improve and a meaningful fraction regress, a pattern the Live UI exposes at trajectory granularity.

The system efficiency numbers are similarly grounded. Agent execution accounts for the overwhelming majority of per-trial wall-clock time. Asynchronous scheduling completes seven training steps over the same wall-clock window where synchronous execution completes three. Lazy image delivery substantially reduces cumulative network traffic and disk writes across the evaluated task images, with gains largest when workloads access only a small fraction of the image.

What Lego-RL demonstrates, more than any single benchmark number, is that the gap between a harness that works for inference and one that works for training is large, specific, and addressable—but only if you instrument the right layers.