OCC4M: Object-Centric 4D Memory for Spatiotemporal Reasoning in Long-Horizon Manipulation

Summary

Long-horizon manipulation often requires reasoning about state absent from the current view, such as a vanished object's location, temporal identity, or the contents of a shuffled container.

Abstract Summary

Long-horizon manipulation often requires reasoning about state absent from the current view, such as a vanished object's location, temporal identity, or the contents of a shuffled container. We present OCC4M ("Occam"), an object-centric 4D memory that maintains persistent tracks in a shared world frame and explicitly represents temporal, motion, and containment relations. A vision-language model (VLM) queries this structured memory to select actionable targets for history-free low-level execution. Across seven simulation conditions and 350 episodes, OCC4M achieves 96.6% memory success and 88.9% end-to-end success, versus 54.6% and 57.7% for FrameSamp, a raw-history VLM baseline using Gemini 3.7 Flash with the complete observation history and the same executor. In a controlled viewpoint-transfer test, OCC4M maintains 100% memory and 98% end-to-end success after a viewpoint change, while full-history FrameSamp falls to near-zero success. On 20 fixed-camera Franka episodes, OCC4M reaches 85% joint memory accuracy, versus at most 30% for FrameSamp across context sizes from $K=16$ to the complete history, and completes 45% of full two-stage tasks. These results support explicit object-centric memory for persistent spatiotemporal reasoning in long-horizon manipulation. Qualitative videos are available at https://occ4m-sup.github.io/occ4m-supplementary/.

Key Points

Long-horizon manipulation often requires reasoning about state absent from the current view, such as a vanished object's location, temporal identity, or the contents of a shuffled container.
We present OCC4M ("Occam"), an object-centric 4D memory that maintains persistent tracks in a shared world frame and explicitly represents temporal, motion, and containment relations.
A vision-language model (VLM) queries this structured memory to select actionable targets for history-free low-level execution.
Across seven simulation conditions and 350 episodes, OCC4M achieves 96.6% memory success and 88.9% end-to-end success, versus 54.6% and 57.7% for FrameSamp, a raw-history VLM baseline using Gemini 3.7 Flash with the complete observation history and the same executor.
In a controlled viewpoint-transfer test, OCC4M maintains 100% memory and 98% end-to-end success after a viewpoint change, while full-history FrameSamp falls to near-zero success.

Method

Perception and geometry SAM 3 detections + calibration Object-centric 4D memory World tracks, times, motion, containment Memory-based target selection Goal + JSON object history One skill-sequence prediction Current-view grounding Reproject target + frozen VLA skills Classical base control updates JSON state selected targets live location new observations Figure 1: Persistent memory as an action interface.

Perception and calibration update object tracks in a shared frame.

Method — OCC4M: Object-Centric 4D Memory for Spatiotemporal Reasoning in Long-Horizon Manipulation — Figure 2: T1c viewpoint-transfer example. The green cube disappears, the base translates $0.30\,\mathrm{m}$, and the robot must place the red cube at the green cube’s remembered world location from the new viewpoint. Yellow ring: OCC4M prediction; pink ring: FrameSamp (Gemini 3.7 Flash) prediction; green ring: ground-truth target.

Experiments

We evaluate a single-arm mobile Panda in ManiSkill3 [ 10 ] , using egocentric shoulder and wrist RGB cameras. Each episode has a scripted observation phase followed by a manipulation query requiring historical information. We evaluate seven conditions spanning six task families (Table 1 ) on the same $N=50$ evaluation seeds.

Each observation phase is recorded once and replayed to every method, providing identical RGB histories and goals; the same recorded camera poses are used for geometric conversion.

Experiments — OCC4M: Object-Centric 4D Memory for Spatiotemporal Reasoning in Long-Horizon Manipulation — Figure 3: Real-world compositional evaluation. The robot must first identify and pick the cup covering the red cube after the cups are shuffled, and then pick the cup occupying the white mug’s remembered initial location. The yellow rings are rendered by OCC4M into the visual input of the low-level policy to indicate the target.

Results

Across the seven conditions, OCC4M achieves 96.6% memory success and 88.9% end-to-end success, compared with 54.6% and 57.7% for FrameSamp given the full $K=32$ observation history (Table 2 ). The raw-history baseline is competitive when the relevant state can be recovered directly from the recorded observations: it reaches 100% memory success on T1 and T5 and 80% on T2.

Sources and Demonstrations

Figures are reproduced from Jedlicki et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.