RoboMemArena: A Comprehensive and Challenging Robotic Memory Benchmark

Summary

Memory is a critical component of robotic intelligence, as robots must rely on past observations and actions to accomplish long-horizon tasks in partially observable environments.

Abstract Summary

Long-horizon robotic tasks require agents to remember and reason about past observations and actions. This paper presents RoboMemArena, a comprehensive benchmark suite designed to evaluate robotic memory across multiple dimensions including spatial memory, temporal reasoning, object permanence, and ... The method demonstrates significant improvements over existing approaches, providing both theoretical insights and practical benefits for real-world deployment. Comprehensive experiments validate the effectiveness of the proposed approach across diverse scenarios and task settings.

Key Points

Proposes RoboMemArena, a novel approach for perception in robotics.
Addresses key limitations in existing methods through innovative architecture design.
Demonstrates strong empirical results on standard benchmarks and real-world evaluations.
Introduces novel training methodology that improves generalization and sample efficiency.
Provides comprehensive analysis of failure modes and ablation studies.

RoboMemArena

In this section, we introduce RoboMemArena , a complex and challenging robotic memory benchmark, in four parts: (1) We present a task suite with four memory-demand categories ( i.e. , transferring, occlusion, counting, and sequential execution) as well as paired real-world tasks ( Section 3.1 ).

(2) We propose a data generation pipeline that combines VLM-based task decomposition, autonomous execution, and multi-conditioned keyframe extraction ( Section 3.2 ). (3) We compare our RoboMemArena with existing robotic benchmarks ( Section 3.3 ).

RoboMemArena — RoboMemArena: A Comprehensive and Challenging Robotic Memory Benchmark — Figure 1 : Visualization of 4 task categories of RoboMemArena. Each row shows the task instruction, subtask decomposition, and execution rollout for Multi-Object Counting, Occlusion, Sequence, and Transferring.

PrediMem: Building Hierarchical Memory with Predictive Coding

We introduce PrediMem , a hierarchical Mem ory framework with Predi ctive coding for embodied memory. It consists of a high-level planner (System 2, denoted by S2), a low-level execution policy (System 1, denoted by S1), a keyframe-grounded memory bank, and an auxiliary predictive coding head.

As shown in Figure 3 , the memory bank $\mathcal{M}_{t}$ combines a long-term keyframe buffer $\mathcal{M}_{t}^{\mathrm{key}}$ with a recent sliding window with fixed horizon $\mathcal{M}_{t}^{\mathrm{rec}}$, i.e.

PrediMem: Building Hierarchical Memory with Predictive Coding — RoboMemArena: A Comprehensive and Challenging Robotic Memory Benchmark — Figure 3 : The PrediMem pipeline. The pipeline comprises two asynchronously coupled components: S1, a low-level action policy that executes the current subtask, and S2, a high-level planner that predicts keyframes and dispatches the next subtask. The predictive coding head path is training-only.

Experiments

We evaluate RoboMemArena and the PrediMem framework around five questions: Does RoboMemArena expose a memory gap in existing VLAs, and can PrediMem close it? Does the end-to-end trained robot memory system surpass powerful closed-source agents? How much do the predictive coding head and the keyframe bank contribute, and how does predictive coding shape the learned memory representations?

How does the scaling of memory influence model performance? How do different baselines perform in the real-world evaluation of RoboMemArena ? Baselines.

Experiments — RoboMemArena: A Comprehensive and Challenging Robotic Memory Benchmark — Figure 4 : Analyses of memory behavior. (a) and (b) shows the sensitivity of average CSR to the recent-buffer size and keyframe-bank capacity. (c) is a t-SNE view showing that predictive coding yields tighter, more discriminative keyframe clusters.

VLM Input Prompt for Data Generation

We use a VLM input prompt template to generate long-horizon robot data by decomposing a coarse task description into executable subtasks and matching each subtask to a predefined planner. Below we show a representative input-only example. System Prompt. You are a robotic data-generation assistant for long-horizon, memory-dependent manipulation tasks. You will be given: 1. One RGB image of the current environment. 2.

VLM Training Prompt and JSON Format

We train S2 with a fixed multi-image prompt template. Below we show a representative Task 1 prompt template adapted from swift_compiled_data.jsonl . System Prompt. You are a robotic planning assistant specialized in memory-based task understanding. Your task is to infer the current primitive action from multi-image observation: 1. Historical keyframes from earlier in the same trajectory. 2.

A recent 5-timestep visual window ending at the current frame. Important rules: - Historical keyframes are earlier than the current window.

Asynchronous Inference Protocol

Given instruction $\ell$ and observation $o_{0}$, S2 runs asynchronously on the recent-frame buffer and memory bank $\mathcal{M}_{t}$, emitting subtask $c_{t}$ and keyframe decision $k_{t}$; the buffered subtask is overwritten whenever a newer S2 result arrives.

Memory-Dependent Subtask Ratio Annotation

We define a subtask as memory-dependent when the correct current subtask cannot be determined from the current observation alone and requires more information from earlier subtasks or observations. Equivalently, a subtask is counted as memory-dependent if removing the execution history or observation would make the correct high-level decision ambiguous.

Reactive Policy Failure Modes

The quantitative gap between reactive and memory-augmented policies maps to two concrete failures. First, the reactive policy cannot distinguish whether a drawer has already been checked once the visual state resets. Second, it cannot preserve the instruction-level constraint that all drawers must be opened before the final placement. Detailed failure modes are provided in Table S3 .

Sources and Demonstrations

Figures are reproduced from Lei et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.