ClawGym II: Exploring Black-Box RL on Agent Harness

Training a language model through a production agent harness like Claude Code or OpenClaw is harder than it sounds. The harness is opaque: it manages context, retries failed tool calls, delegates to subagents, and compacts long histories—none of which the RL algorithm can see. What the training loop observes is a scattered set of model calls at the serving boundary, not a clean trajectory. ClawGym II builds the infrastructure to turn that mess into a working RL signal.

The core mechanism is a serving proxy placed at the model boundary. Every request the harness makes passes through it; the proxy records exact input tokens, generated tokens, and log-probabilities without touching the harness internals. Those captured calls are then organized into prefix trees, which reconstruct the shared interaction structure across a rollout's branching calls. Root-to-leaf paths through the tree become candidate training trajectories. Dead leaves from retries get pruned; rollouts that branch excessively—a sign of repeated failures rather than legitimate subagent work—get discarded entirely.

Both PPO and GRPO are adapted to optimize over this tree structure. For GRPO the adaptation is natural: advantages are normalized within rollout groups per task–harness pair, and shared prefix nodes count once toward the loss. For PPO the adaptation is more of a simplification—sibling trajectories are treated independently, which the paper acknowledges increases advantage variance and leaves a more principled treatment to future work.

Training–inference consistency gets two explicit fixes. A token-in-token-out discipline keeps the harness's reformatted view of model outputs strictly separate from the token record used for training, so harness normalization never corrupts the gradient target. A token-level importance-sampling correction then accounts for the numerical gap between the inference engine's recorded log-probabilities and the training engine's recomputed ones.

The empirical results use Qwen3-30A3B as the backbone. Training through OpenClaw improves Pass@1 on ClawGym-Bench by roughly 10 points over the initialization; training through Claude Code improves it by roughly 15 points. Both runs stay stable across 200–400 optimization steps. A white-box agent loop trained with the same RL algorithms scores higher when evaluated in its own loop, but transfers less well to OpenClaw than the black-box-trained model does—suggesting harness-specific interaction patterns are genuinely learned and not just a side effect of stronger general capability.

Mix-harness training, where a single model is jointly optimized through both OpenClaw and Claude Code in the same run, matches or slightly beats the single-harness models under each respective harness. The framework also extends without pipeline changes to JobBench and OfficeQA, two structurally different task distributions, with consistent score improvements in both cases.

The honest limitation the paper names is that subagent and compaction trajectories are excluded from optimization entirely—they receive the same terminal reward as the main trajectory but their contribution to the task outcome is ambiguous, so they are dropped. That is a real gap: in long-horizon tasks, subagent work can constitute most of the actual execution.