What Should Embodied Agents Remember? Benchmarking and Evaluating Memory Modality for Embodied Agents

Junxiang Wang, Yuchen Liu, Yi Wu, Guofei Chen, Nir Pechuk, Tao Chen, Zackory Erickson, Yuzhe Qin, Tianyu Li · N/A · 2026

Summary

Memory allows large language model (LLM) agents to use past experience, often stored as textual records, when responding to requests.

Abstract Summary

Memory allows large language model (LLM) agents to use past experience, often stored as textual records, when responding to requests. For embodied agents that control robots, visual and spatial observations introduce a broader choice of memory modalities, affecting both the information available for subsequent tasks and the model inference costs of constructing and using memory. We introduce MeMoREval, a benchmark for studying this problem through a controlled experience-request protocol. Agents first experience a sequence of events without knowing the subsequent tasks, and subsequently they use memory to respond to information and action requests without receiving new observations. The benchmark comprises 53 tasks across four simulated indoor environments and measures task success alongside the inference costs of memory construction and use. Our reference evaluations reveal that storing frames outperforms storing generated descriptions at lower write-time inference cost for the primary model, while storing frames with scene graphs achieves the highest aggregate task success for every tested model. Extensions to accumulated experience, selective frame retention, and visual input to memory writers demonstrate how the benchmark accommodates further memory-design investigations. Two physical-robot demonstrations establish the feasibility of applying its protocol beyond simulation.

Key Points

Memory allows large language model (LLM) agents to use past experience, often stored as textual records, when responding to requests.
For embodied agents that control robots, visual and spatial observations introduce a broader choice of memory modalities, affecting both the information available for subsequent tasks and the model inference costs of constructing and using memory.
We introduce MeMoREval, a benchmark for studying this problem through a controlled experience-request protocol.
Agents first experience a sequence of events without knowing the subsequent tasks, and subsequently they use memory to respond to information and action requests without receiving new observations.
The benchmark comprises 53 tasks across four simulated indoor environments and measures task success alongside the inference costs of memory construction and use.

Preliminaries: LLM-Based Embodied Agents and Memory

We consider LLM-based embodied agents in which a language or vision-language model selects physical skills for a robot. The interaction loop resembles that of a coding agent: the model receives a task and observations, calls a tool, and uses the returned feedback and updated observations to decide the following action.

Each tool corresponds to a physical skill, such as navigating to a room, grasping an object, or opening a drawer, executed by a separate robot controller so the model chooses actions without generating motor commands.

Preliminaries: LLM-Based Embodied Agents and Memory — What Should Embodied Agents Remember? Benchmarking and Evaluating Memory Modality for Embodied Agents — Figure 1: Overview of our embodied agent memory evaluation protocol, on a real robot. During the experience phase, the agent takes in an instruction (1) and executes corresponding actions (2) while writing into its memory (3). During the request phase, the agent receives another instruction (4) and reads its memory (5) for relevant information in order to perform the correct actions (6).

MeMoREval: Evaluation Protocol and Benchmark

We introduce MeMoREval, a benchmark for evaluating memory modalities in robotic embodied agents. We first describe its evaluation protocol, including task composition and measured performance metrics, such as task success and inference costs (§ 4.1 ), and then discuss the simulation environments and tasks that instantiate it (§ 4.2 ).

Experiments

Memory conditions. Our main comparison varies the evidence archived for later retrieval while holding the memory writer’s textual input fixed. For a given backend, task, and seed, the six main conditions share the textual memory constructed from the common observations in § 4.2 . The text condition retains only this memory.

The frames condition additionally archives the annotated camera frames, while frames+graph also archives scene graph snapshots.

Experiments — What Should Embodied Agents Remember? Benchmarking and Evaluating Memory Modality for Embodied Agents — Figure 3: (a) Task success against per-experience-step write cost. (b) Question success against per-information-request read cost. (c) Action success against per-action-request read cost. Aggregated across all tasks, memory backends, and three seeds, for Gemini 3.7 Flash. The transcript backend is excluded from (a) as its writer is not LLM-based. Error bars show the range across seeds.

Sources and Demonstrations

Figures are reproduced from Wang et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.