Robots don't need growing context windows to remember — fixed-size memory wins

Most robot manipulation policies suffer from a specific kind of amnesia: each action call sees only the current frame and language instruction, discarding everything that came before. This works fine when the scene is fully observable, but fails the moment a robot needs to remember where it picked something up, or which battery orientation it already tried. MemBodied attacks this directly with a fixed-capacity episodic memory that never grows, regardless of episode length.

The architecture has two parts. An associative state stores layer-wise matrices updated after each policy call using a gated delta rule — the write corrects what is already associated with a key rather than blindly accumulating values. The second part is an episode anchor: a compact spatial reference to the first observed scene, retrieved via cross-attention at every subsequent call. The anchor exists because repeated delta-rule updates tend to overwrite fine-grained early details; it preserves them without appending the raw first frame to the policy context.

The write schedule is deliberately causal. After an action chunk executes, the model combines the action summary with the visual consequence of that action and writes the pair into memory. Stored values therefore represent transitions — what was done and what resulted — rather than isolated snapshots.

On five RMBench tasks designed around memory complexity, MemBodied reaches 50.0% mean success, beating a stateless baseline by 43.6 percentage points and a vanilla recurrent baseline by 33.2 points. The compressed-history method NativeMEM reaches 38.4% in the same setup. More striking is the efficiency gap: MemBodied runs with 91.9% lower inference latency and 9.5% less peak GPU memory than NativeMEM, while adding only about 40 million parameters — roughly one-tenth of NativeMEM's parameter overhead.

The gains are not purchased at the cost of general performance. On LIBERO-Long, MemBodied hits 90.6%, a 5.4 percentage point improvement over π0's 85.2%. The four-suite LIBERO mean stays comparable at 95.1% versus π0's 94.2%.

Manual annotation of 50 paired rollouts per task confirms that improvements are localized to exactly the decisions that require past information. On Put Back Block, wrong-pad returns drop from 37 out of 50 to 13 out of 50; the preceding pick-and-place stages are identical between policies. On Swap Blocks, premature button presses fall from 41 to 19. Remaining failures are mostly motor execution errors — stalling, dropped objects, missed button presses — not memory failures.

One honest limitation surfaces in the ablations: the anchor hurts on Block Ranking, a task that depends on tracking evolving progress rather than recalling a fixed initial state. A fixed first-frame reference is simply less useful when the relevant information is cumulative rather than static. The paper flags combining explicit video history with associative memory as unresolved; adding NativeMEM's tokens alongside MemBodied improves NativeMEM's standalone score but still trails MemBodied alone, suggesting the two representations interfere as much as they complement.