Solving the math before building the environment fixes agentic RL's reward problem

The central problem VHD-Play addresses is a sequencing flaw in how agentic training environments get built. Most generation approaches fix the interactive environment first, then figure out how to evaluate it afterward—leaving dynamics and reward signal to be reconciled post hoc. VHD-Play inverts this: sample and solve a mathematical model first, then render its decision process as a stateful environment. The solution is already in hand before any environment exists, so the outcome signal is never retrofitted.

The mechanism is concrete. A family-specific solver—dynamic programming for inventory replenishment, Held-Karp recursion for routing, HiGHS for linear programs—computes the reference optimum before any language model touches the instance. A frozen setter then receives a real-world corpus passage and the full parameter draw, and generates scenario text, a relational database, and executable tools. The player sees none of the underlying parameters; it must probe, commit, and navigate state changes to recover what it needs. The score is arithmetic over realized utility and two fixed references, with no learned judge anywhere in the path.

At 3,300 admitted environments costing a few cents each, the substrate is cheap to extend. Training Qwen3.6-35B-A3B on three mechanism families lifts its mean agentic score from 0.204 to 0.815 across a five-family diagnostic. Gains appear on held-out instances from all three training families, eight unseen families, and on external benchmarks the training environments share nothing with. On a 365-day e-commerce benchmark, the trained checkpoint completes all five storefront runs without bankruptcy and posts a mean ending balance above the Qwen3.7-Max reference. Ten interaction-focused BFCL V4 cells are evaluated, with eight improving and two unchanged.

The written-out versus agentic comparison is the paper's sharpest finding. The base model scores near ceiling when given the complete problem in a prompt, but falls sharply when the same mechanism unfolds through stateful tools. Crucially, most of that gap persists even when all parameters are revealed upfront in the agentic form—the model can read the numbers but still fails to plan across the horizon. In a seven-period allocation task, the base policy writes 'For each period, find best combination,' spends 27 of 29 shared-budget units in the first three periods, and scores 0. The trained policy constructs one horizon-wide allocation and scores 1. The failure is not arithmetic; it is the inability to hold commitments across time.

The co-scaling experiment is worth attention. A frozen copy of the same 35B checkpoint acts as setter, generating environments its own base policy cannot yet navigate. As mechanism size grows from 6 items over 5 periods to 11 items over 11 periods, scale-matched training gains increase rather than shrink—the two larger configurations average higher gains than the two smaller ones. A model can construct a harder environment than it can initially solve, and that gap remains learnable as complexity grows.