Miles v0.1 treats train-rollout mismatch as a first-class engineering problem

Training a 744-billion-parameter mixture-of-experts model with fully asynchronous RL on 64 GPUs is the concrete claim Miles v0.1 makes, and the paper spends considerable effort explaining why that is harder than it sounds.

The central problem is that rollout and training want to disagree. Different kernels, different precision formats, different batching shapes — each one shifts the log-probabilities the trainer computes away from what the rollout engine actually sampled. In a mixture-of-experts model this gets worse: the router can send the same token to different experts on each side, and the paper cites prior work showing that uncorrected routing discrepancy can end in catastrophic training collapse. Miles addresses this with two mechanisms. The TITO session server owns tokenization across every turn of a multi-turn episode, so the trainer sees the exact token IDs the engine produced rather than a re-serialized approximation. The R3 replay mechanism records which experts each token visited during rollout and replays those assignments during the training forward pass, at a cost of roughly 60 MB per 32K-token trajectory at 60 layers.

Fully asynchronous execution is the other load-bearing idea. Rather than alternating generation and training phases, Miles runs them on separate GPU pools simultaneously, with a bounded data buffer between them. The buffer is the single point where staleness is enforced: a group's staleness is defined as the current trainer weight version minus the oldest version appearing anywhere in that group, a deliberately pessimistic definition. When the buffer fills, generation blocks; when it empties, training stalls — the paper gives explicit metrics for detecting which side is the bottleneck.

The reference run is a 100-step agentic coding experiment: GLM-5.2 744B-A40B solving terminal-bench-2 tasks, 32 GPUs generating and 32 training, each trajectory an agent at a command line inside a per-episode sandbox. Median step time is 263 seconds. The prefix-cache hit rate holds at 96% through affinity routing. The log-probability divergence between rollout and trainer averages 0.0369 across the run. Task reward rises from 0.438 to 0.556 on a nine-step moving average, though the paper explicitly declines to call this a measured improvement given it is a single run on a single task distribution.

The paper is unusually candid about its own limits. True-on-policy alignment, which pins both engines to bitwise-identical kernels, currently covers only dense Qwen3 0.6B and 4B variants. P2P weight transfer is slower than broadcast on single-node deployments by up to about 70%. Vision-language models cannot use the session server yet. Several recipes exist only as open pull requests. The evidence-level taxonomy — fully gated, proxy gated, verified, not verified — is applied per recipe rather than per model family, which is a more honest accounting than most systems papers attempt.