SPADE: Self-Play in Adaptive Synthetic Executable Environments

The core problem SPADE addresses is simple to state: fixed training environments stop teaching once a model outgrows them. Every existing approach—hand-curated pools, statically synthesized tasks, frozen verifiers—shares this flaw. The environment distribution stays put while the learner moves on.

SPADE's answer is to make environment design itself a learnable role. A single LLM alternates between two jobs: writing complete training environments as executable Python code with a Gym-style reset/step interface, and acting as the agent that learns inside them. Both roles share the same weights and update through the same GRPO loop.

The mechanism that keeps difficulty calibrated is hint-based regret. The Environment Designer writes a privileged hint alongside each environment—a partial solution sketch or key structural observation. The Designer's reward is the gap between the Reasoning Agent's return with and without that hint. High regret means the task is solvable with guidance but not yet without it: exactly the learning frontier. Mastered tasks and impossible tasks both produce low regret and get deprioritized. The paper proves that at a pure Nash equilibrium of this game, the hint gap collapses to zero everywhere—meaning the Reasoning Agent becomes hint-free optimal on every valid environment.

Two components prevent the Environment Designer from collapsing onto repetitive output. Corpus grounding seeds each generation from a freshly sampled document—mathematics and science text for the games setting, code and API documentation for tool use. Without it, the no-corpus ablation emits the same rotating-maze task 41 times in a row across steps 290–312. An environment memory buffer, annotated with regret scores, steers the Designer away from already-mastered or intractable territory each round.

At 30B-A3B scale, SPADE improves over the strongest fixed-environment baseline across eight held-out benchmarks spanning math, science, code, and procedural reasoning. The tool-use setting lifts ACEBench-Agent and BFCL v4 multi-turn, with the largest gains where benchmark task structure most closely mirrors the generated environments. Crucially, the margin over fixed baselines grows with model size: larger models exhaust static pools faster and benefit more from an adaptive curriculum.

The ablations are instructive. Freezing the Environment Designer—even substituting GPT-5.5 as the frozen designer while keeping corpus grounding and memory—recovers only a fraction of SPADE's gain and peaks early around step 175. Removing corpus grounding collapses environment diversity to near-zero. The gains require co-adaptation: the Designer and Agent must train together.

The paper is honest about limits. Environment complexity is bounded by what the base model can express in context. The learning rule itself is fixed and human-authored—SPADE does not modify GRPO. And hint-based regret, while motivated by minimax regret theory, carries no formal optimality guarantee for the actual curriculum produced.