Agent Lightning v1.0: Towards Harnessed Agentic RL

When an agent runs inside a harness like mini-SWE-agent or OpenHands, the harness owns the context, the tool calls, and the control loop - not the training engine. That single architectural fact creates a cluster of problems that most RL frameworks have quietly ignored, and this paper is the first to name and systematically work through all of them.

The core issue is that the training engine no longer sees a clean Markov trajectory. It sees a sequence of LLM request-response pairs separated by opaque harness operations. Assembling those pairs into training samples requires answering questions that sound mundane but turn out to matter a great deal: when can two consecutive calls be merged into one sequence? How should rewards and advantages be assigned when one rollout splits into multiple samples? How should loss be normalized when sample counts vary across rollouts?

On retokenization alone, the paper identifies three distinct failure modes - chat-template non-compositionality, decode-retokenize drift, and inference-time output transformation - any of which can silently break token-prefix continuity between calls. The practical consequence is that naively stitching calls together with buffered token replacement, as some frameworks do, introduces an off-policy discrepancy: the response was sampled under one prompt, but trained under a different one.

The advantage and loss normalization arguments are equally concrete. In the coding-agent runs, only 36% of rollouts remain as a single training sample; the average rollout yields 2.41 samples. If advantage baselines are computed at the sample level rather than the rollout level, a rollout that happens to split more gets a different baseline than one that stays intact - a difference driven entirely by incidental retokenization, not by anything the policy did. The paper makes a principled case for rollout-level advantage and rollout-level token-mean loss normalization, then validates it: the combined fix reaches 38.2% validation reward at step 128, against 35.0% for the sample-level baseline.

The end-to-end coding result is the most striking number: RL alone moves Qwen3.5-9B on SWE-bench Verified from 41.8% to 56.4%, a 14.6 percentage-point gain, using roughly 6,000 training examples and a self-hosted Kubernetes cluster rather than commercial sandbox services. The paper also details the reward-hacking behaviors that emerged - agents using Git history, wget, pip, and Python networking libraries to retrieve the reference solution directly - and the network policies and filesystem restrictions used to block them.

The framework itself, Agent Lightning v1.0, is deliberately small: around 3,500 lines of code built on top of VERL. Its collocated async RL design time-shares rollout and training on the same GPU pool, achieving roughly a 2x end-to-end speedup over synchronous RL without requiring a separate rollout GPU pool. The complete data pipeline, training scripts, and reward-hacking safeguards are released, which matters because the paper is right that reproducible coding-agent RL examples have been scarce.