τ_0-VLA: a Hierarchical Robot Foundation Model with World-Model-Guided Test-Time Computation

Long-horizon robot manipulation fails not because robots lack motor skill but because they make consequential planning decisions with a fixed compute budget, regardless of how ambiguous or difficult those decisions are. τ₀-VLA addresses this directly by treating high-level subtask generation as a compute-scalable inference problem rather than a single forward pass.

The architecture is hierarchical in a specific and meaningful way. A high-level policy maintains an execution memory, generates candidate subtasks, and decides—based on token-confidence statistics from its own forward pass—whether to commit immediately or invoke a search procedure. That search follows a propose–predict–evaluate loop: a proposal model generates candidate subtasks, a world model predicts the terminal visual state each candidate would produce, and a value model scores those imagined outcomes. Beam search expands promising branches recursively, and a reflective model then generates the final subtask from the retained branches. Crucially, the reflective model's output is not constrained to the candidate set—it can synthesize something new from the evidence. The low-level policy then executes whatever subtask the high-level policy commits to, across a unified 40-dimensional state and action space that covers fixed-base, bimanual, and mobile whole-body control without embodiment-specific output heads.

The low-level policy is trained on a large heterogeneous corpus of real-world robot data drawn from multiple embodiments, supplemented with multimodal vision-language co-training. The high-level policy's supervision is derived automatically from existing annotations, with no additional per-sample human labeling required. A notable design choice is the memory-perturbation training: the system is explicitly trained to repair execution memories that lag behind, run ahead of, or otherwise misrepresent actual task progress—which matters enormously for tasks where a single observation cannot reveal whether a step has already been completed.

The seasoning problem in Tomato and Egg Stir Fry illustrates why this matters. Adding salt produces almost no visible change, so a policy relying only on the current observation tends to add salt repeatedly or skip it. The hierarchical system resolves this by recording seasoning progress in memory. Similarly, in the out-of-distribution Book Organization setting—where initial book arrangements were never seen during training—test-time computation achieves substantially higher next-subtask prediction accuracy than both Plan Once and the Best-of-k baseline, which evaluates candidates with the same world and value models but without multi-step expansion or reflective commitment.

The compute–accuracy relationship follows a saturating curve: gains are steep at low budgets and taper off as more computation is allocated. This is a practically useful property—moderate inference budgets capture most of the benefit without requiring unbounded search.

Evaluation tasks run up to 12 minutes per episode and include room cleaning, meal preparation, tea making, and laundry collection—tasks with up to 25 annotated milestones—across three different robot platforms. Results show that test-time computation improves both open-loop subtask prediction accuracy and closed-loop task success rates across all evaluated tasks, with gains most pronounced where fixed execution plans are insufficient and where intermediate states are visually ambiguous.