VibeWorlding: Can Multimodal Agents Construct 3D Open Worlds End-to-End?

Collision-free 3D spatial editing is where every current multimodal agent breaks down. That is the central finding of VibeWorlding, a framework that formalizes the problem of building interactive 3D open worlds from natural-language queries and then measures how badly today's best models fail at it.

The benchmark, VWE-Bench, is built from over two thousand high-quality 3D assets spanning twenty semantic categories, more than three hundred human-annotated seed worlds, and nearly seven thousand reverse-synthesized queries covering both from-scratch construction and multi-turn refinement. Queries range from a bare theme like "an eerie wasteland shrine" all the way to full spatial blueprints, and a small distractor subset embeds physically impossible or library-absent requests to test whether agents clarify rather than comply blindly.

The headline result is blunt: even GPT-5.5 and Qwen3.8-Max fall below a sixty percent overall Pass@1 on VWE-Bench. Untrained open models sit far lower. The six-capability breakdown explains why. Collision avoidance is the weakest dimension across every model tested, scoring in a narrow band that barely moves even after reinforcement learning. The failure mode is specific: agents understand what the user wants and select the right asset, but emit wrong 3D coordinates—often the correct displacement magnitude in the precisely wrong direction, a coordinate-frame confusion rather than a magnitude error. A second failure is scope violation, where the agent correctly executes a requested edit but silently deletes or moves assets the user never mentioned.

The training pipeline, VibeWorlding-Gym, addresses this through a two-stage recipe. Cold-start supervised fine-tuning on reverse-synthesized trajectories installs basic physical and ecological competence. Multimodal RL with GRPO then drives the gains that matter most: 3D reasoning improves substantially, and the agent learns to use rendered multi-view feedback—five camera angles returned after every tool call—to verify its own edits rather than operating blind. The reward signal is a dual-constraint verifier combining geometric collision and height checks with an MLLM judge for intent fulfillment, ecological plausibility, and retrieval quality. Crucially, verified queries carry a deterministic ground-truth reward, and the paper shows these produce smoother, more reliable learning curves than the MLLM-as-judge path used for open-ended queries.

The result is that an eight-billion-parameter model post-trained from Qwen3-VL-8B reaches parity with Gemini 3.1-pro, and the thirty-billion mixture-of-experts variant outperforms every closed-source frontier on overall Pass@1. The gains are sharpest on the verified track, where precise rule-checkable editing is required—exactly the dimension where frontier models remain most deficient.

One honest limitation the paper surfaces: the asset library is uniformly cartoon or low-poly in style, built through an internal Tencent crowdsourcing pipeline, and the most complex seed worlds contain only a few hundred placed assets. Truly open-ended world construction at city scale is not yet within reach of this framework. The authors also note that rendering through Blender at every turn is slow, which constrains both evaluation throughput and RL data scaling. These are real constraints, not caveats—the benchmark is a controlled environment, and the gap between it and a production game engine remains wide.