Spatial Memory for Out-of-Vision Manipulation in Vision-Language-Action

Spatial Memory for Out-of-Vision Manipulation in Vision-Language-Action

Pengteng Li, Weiyu Guo, He Zhang, Tiefu Cai, Xiao He, Yandong Guo, Hui Xiong · OPPO Research, BUPT, USTC · 2026

Summary

SOMA introduces an explicit spatial memory framework that enables VLA models to track and manipulate objects even after they leave the camera's field of view.

Abstract Summary

We introduce SOMA, the Spatial Memory framework for Out-of-Vision Manipulation in Vision-Language-Action (VLA) models. Most existing VLAs implicitly assume that task-relevant objects are always visible, leading to brittle and reactive behaviors when targets fall outside the camera's field of view. SOMA augments VLAs with an explicit spatial memory buffer that tracks object locations even after they leave the camera frustum, enabling robust manipulation in cluttered environments with frequent occlusions. The framework stores object-centric visual features alongside their egocentric poses at the moment of detection. When an object exits the field of view, its memory representation persists in a learned token buffer. A lightweight cross-attention mechanism queries this buffer during action generation, retrieving only the tokens relevant to the current language instruction. This design enables the policy to react to objects it can no longer see, a common scenario in tabletop manipulation when arms or other objects block the camera. SOMA is evaluated on both simulated and real tabletop manipulation tasks with a 7-DoF Franka arm. Experiments show that SOMA significantly improves success rates on out-of-vision manipulation tasks—such as placing an object behind another or retrieving an item from a drawer that occludes the target—while maintaining baseline-level performance on standard in-view tasks. The memory mechanism adds negligible latency and can be inserted into existing VLA architectures without retraining the entire backbone, making it a practical plug-in for deployed systems.

Key Points

  • Introduces SOMA, a spatial memory augmentation for VLA models.
  • Addresses occlusion and out-of-field-of-view manipulation failures.
  • Uses object-centric memory buffer with cross-attention retrieval.
  • Validated on simulated and real robot manipulation tasks.
  • Maintains in-view performance while improving out-of-vision success rates.

Additional Notes

Overview

  • Octo: An Open-Source Generalist Robot Policy
  • RT-2: Vision-Language-Action Models
  • OpenVLA: An Open-Source Vision-Language-Action Model
Share

Related Papers

DynaFLIP: Rethinking Robotics Perception via Tri-Modal-Dynamics Guided Representation

DynaFLIP: Rethinking Robotics Perception via Tri-Modal-Dynamics Guided Representation

Jusuk Lee, Seungjae Lee, Jonghun Shin et al. · arXiv preprint · May 2026

Robot manipulation critically depends on perception that preserves the action-relevant aspects of a scene. Yet most robot learning pipelines are built upon visual encoders pre-trained for static recog

foundation-models perception manipulation representation-learning vla
PDF Intermediate
No code repo Code updated: May 2026