Robotics Research Hub
Spatial Memory for Out-of-Vision Manipulation in Vision-Language-Action

Spatial Memory for Out-of-Vision Manipulation in Vision-Language-Action

Pengteng Li, Weiyu Guo, He Zhang, Tiefu Cai, Xiao He, Yandong Guo, Hui Xiong · OPPO Research, BUPT, USTC · 2026

vla manipulation foundation-models
PDF arXiv

Summary

SOMA introduces an explicit spatial memory framework that enables VLA models to track and manipulate objects even after they leave the camera's field of view.

Abstract Summary

We introduce SOMA, the Spatial Memory framework for Out-of-Vision Manipulation in Vision-Language-Action (VLA) models. Most existing VLAs implicitly assume that task-relevant objects are always visible, leading to brittle and reactive behaviors when targets fall outside the camera's field of view. SOMA augments VLAs with an explicit spatial memory buffer that tracks object locations even after they leave the camera frustum, enabling robust manipulation in cluttered environments with frequent occlusions. The framework stores object-centric visual features alongside their egocentric poses at the moment of detection. When an object exits the field of view, its memory representation persists in a learned token buffer. A lightweight cross-attention mechanism queries this buffer during action generation, retrieving only the tokens relevant to the current language instruction. This design enables the policy to react to objects it can no longer see, a common scenario in tabletop manipulation when arms or other objects block the camera. SOMA is evaluated on both simulated and real tabletop manipulation tasks with a 7-DoF Franka arm. Experiments show that SOMA significantly improves success rates on out-of-vision manipulation tasks—such as placing an object behind another or retrieving an item from a drawer that occludes the target—while maintaining baseline-level performance on standard in-view tasks. The memory mechanism adds negligible latency and can be inserted into existing VLA architectures without retraining the entire backbone, making it a practical plug-in for deployed systems.

Key Points

  • Introduces SOMA, a spatial memory augmentation for VLA models.
  • Addresses occlusion and out-of-field-of-view manipulation failures.
  • Uses object-centric memory buffer with cross-attention retrieval.
  • Validated on simulated and real robot manipulation tasks.
  • Maintains in-view performance while improving out-of-vision success rates.

Method

As shown in Figure 2 , SOMA addresses the challenge of out-of-vision manipulation by equipping VLA with a spatial memory that supports persistent scene understanding beyond the current camera view.

Prior to manipulation, if the target object referenced in the instruction cannot be localized in the current camera observation by the perception module, the robot performs a dedicated scene observation phase using a movable head camera to initialize Spatial Memory Construction .

Method — Spatial Memory for Out-of-Vision Manipulation in Vision-Language-Action
Figure 3 : Illustration of our real world benchmark settings. We design five challenging out-of-vision pick-and-place (PnP) tasks to evaluate the robot’s OOV manipulation capabilities. Tasks (1–3) require the robot to use its left arm to pick up the object located outside the current field of view and place it into the basket. Task (4) involves sequential pick-and-place of the object followed by the other object into the same basket. Task (5) requires dual-arm coordination to pick and place the object into the basket.

Experiments

We conduct extensive real-world experiments to validate the effectiveness of SOMA in OOV manipulation. We further evaluate its memory-centric design on RoboCasa Tabletop GR1 ( Bjorck et al., 2025 ) and SimplerEnv ( Li et al., 2024 ) . Real World OOV Tasks.

As shown in Figure 3 and 5 , we construct a real-world benchmark of five out-of-vision pick-and-place (PnP) tasks with increasing difficulty to evaluate SOMA under partial observability.

Experiments — Spatial Memory for Out-of-Vision Manipulation in Vision-Language-Action
Figure 5 : Illustration of task execution examples for our five challenging out-of-vision tasks in real world using our proposed SOMA.

mpact Statement

This work addresses the challenge of manipulation under limited operational field of view by introducing a spatial memory framework that enables robots to reason about and act on objects beyond the current camera view.

By integrating multi-view scene exploration, persistent spatial memory, and contextual interaction, the proposed approach improves robustness in long-horizon manipulation tasks where relevant objects cannot be directly observed at execution time.

mpact Statement — Spatial Memory for Out-of-Vision Manipulation in Vision-Language-Action
Figure 6 : Illustration of our self-designed robot construction.

Sources and Demonstrations

Figures are reproduced from Li et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.

Share

Related Papers