AgentMercury: Your Agent Can Synthesize Verifiable Environments for Business Scenarios at scale

The core claim here is that you do not need to build training environments around your target benchmarks. AgentMercury constructs environments from high-level business scenarios first, then lets tasks emerge from those worlds rather than the other way around. The distinction matters: task-centric construction couples the world to whatever you are trying to measure, so adding more tasks does not necessarily add more diverse worlds. Scenario-grounded construction inverts this, and the paper argues the resulting diversity is what actually transfers.

The mechanism has three named roles. A "Planet" instantiates a persistent world with entities, services, tools, persistent state, and cross-service invariants expressed as executable verification conditions. A task generator then seeds objectives from that world. The agent interacts through tool calls, and a deterministic grader checks the final database state against both task-specific assertions and world-level invariants. Because the invariants are checked post-episode rather than enforced during execution, the agent is responsible for satisfying cross-service constraints rather than having the simulator do it automatically.

The scale is 4,783 executable environments across 14 industries and 50 countries, yielding a training corpus of 43,300 tasks. A Qwen3.5-4B model trained with GRPO on this corpus moves from 12.3 to 15.7 on EnterpriseOps-GYM and from 45.9 to 56.0 on AIME26, despite neither benchmark appearing in the training environments. Gains extend to competitive programming, scientific computing, and tool-use benchmarks. The checkpoint-level analysis shows these improvements accumulate progressively rather than appearing at a single step, and the degenerate-response ratio stays near zero throughout, which rules out reward hacking through malformed outputs.

The second experiment asks whether environment construction itself can be learned. A base Qwen3.5-35B-A3B model passes the full 12-validator oracle on only 3.3% of held-out business briefs. Fine-tuning on 29,823 construction traces raises that to 83.3%. The training mix covers brief-to-world generation, partial-stage completion, validator-guided repair, and intent-to-diff prediction. One counterintuitive finding: providing a construction recipe to the fine-tuned model actually hurts performance, with 27 of 30 recipe-conditioned generations failing the cross-service check. The construction procedure, once internalized in the weights, is disrupted rather than helped by additional procedural prompting.

The dominant failure mode for zero-shot authoring across API models is collapsing a cross-service invariant so that both its trigger and target land in the same service. This is invisible to surface inspection but caught immediately by the executable oracle, which the paper treats as essential rather than optional infrastructure.

What the paper does not yet close is the feedback loop: environment synthesis is still an offline process that cannot observe where the policy is failing and generate targeted worlds in response. The authors flag this explicitly as the next step.