Coding agents that freeze before deployment already outperform hand-built TAMP planners

Off-the-shelf coding agents—given only a task description, simulator access, and a $20 model budget—synthesize programs that outperform hand-engineered TAMP planners on 28 constrained manipulation environments. That result is worth sitting with. The planners receive hand-written predicates, operators, samplers, and motion skills; the agents receive none of that. Yet across the 16 environments where a planner is available, the best agent configuration (Codex with GPT-6 Astra) averages 95% success against the planner's 47%.

The mechanism matters more than the headline number. These agents are not querying an LLM at test time. Each agent spends its budget interacting with a sandboxed simulator—writing custom tests, probing edge cases, calibrating physical models—and then freezes a single program. That program runs entirely without LLM involvement during evaluation. The synthesis process is agentic; the deployed policy is not. This distinction is what makes the efficiency results credible: main-setting Astra programs average 1.3 ms per action, and on the 14 environments with a planner available and multiple object counts, agent programs average 0.5–2.1 seconds per instance versus 29 seconds for the planners.

The interaction logs are the most interesting part of the paper. In one Shelf run, an Opus agent constructs a kinematic model of a Kinova Gen3 arm from its own prior knowledge, uses a grasped cube as a position marker, and fits six parameters to calibrate mount and grasp offsets—reducing prediction RMSE from 38.9 to 1.8 mm—before using the fitted model for inverse kinematics. In StickButton, the same agent builds a custom test suite by repeatedly calling reset to surface wall-adjacent edge cases, then tests its program against both typical and adversarial configurations. These are not behaviors that look like retrieval from training data; they look like engineering.

The unexpected strategies the agents discover are harder to dismiss than the aggregate numbers. One program ignores the provided sweeping tool and uses the gripper directly. Another rotates and regrasps a scooping tool to increase capacity. The paper notes, correctly, that strategies absent from any published solution are difficult to attribute to memorized pretraining.

Limitations are real. The setup assumes full observability and object-centric states—no perception, no partial information. Dynamic 3D environments involving sweeping or pouring many small objects remain largely unsolved in the main setting. Training data for these models is undisclosed, so prior exposure to benchmark code cannot be ruled out, though the synthesis logs argue against simple recall.

The comparison with LLMGenPlan is telling. That method also receives environment source code and uses the same Opus 5 model with the same $20 budget, but cannot run code or choose its own experiments. It averages 28% success. The agentic loop—the ability to probe, test, and revise—accounts for most of the gap, not model capability alone. Source access helps the agents too, raising Astra's mean from 86% to 95%, but it is clearly not sufficient on its own.

For anyone building agents that need to reason about physical constraints, this is a strong existence proof that the synthesis-then-freeze pattern can work at scale.