Beyond Episodic Evaluation: Memory Architectural Bottlenecks in Sequential Embodied Question Answering

Zikui Cai, Kaushal Janga, Tan Dat Dao, Seungjae Lee, Shivin Dass, Mingyo Seo, Kaiyu Yue, Mintong Kang, Nandhu Pillai, Monte Hoover, Aadi Palnitkar, Ruchit Rawal, Ruijie Zheng, Bo Li, Yuke Zhu, Roberto Martín-Martín, Tom Goldstein, Furong Huang · N/A · 2026

Summary

Embodied question answering (EQA) is traditionally evaluated under an episodic formulation, where agents solve each task independently and reset internal state between episodes.

Abstract Summary

Embodied question answering (EQA) is traditionally evaluated under an episodic formulation, where agents solve each task independently and reset internal state between episodes. However, real-world robots operate continuously and must accumulate, retain, and selectively reuse information acquired from prior interactions. Despite this practical requirement, the architectural mechanisms needed to support sequential memory in EQA remain underexplored. In this work, we investigate how different memory architectures behave when EQA agents are evaluated sequentially, with multiple questions answered in the same scene while memory is carried forward across queries. We find that simply preserving existing memory is often insufficient. Agents that retain only traversability information, such as 2D occupancy maps, remember where the robot has explored but not the visual-semantic evidence needed for later questions. Agents trained on short-horizon episodic data face a different challenge: when exposed to continuous, multi-query histories, their inherited context suffers from severe temporal mismatch, rather than forming a reusable scene representation. To overcome this architectural bottleneck, we highlight the necessity of structured, spatially grounded memory: architectures that map persistent visual observations onto metric 3D geometry preserve visual-semantic evidence in a coherent scene representation. Extensive experiments in simulated environments reveal that this form of memory breaks the accuracy-efficiency tradeoff in sequential settings, simultaneously achieving higher answer accuracy and lower navigation costs. We further validate these findings on a real-world mobile robot, demonstrating that spatially grounded visual memory is critical for enabling continuous, intelligent operation in physical environments.

Key Points

Embodied question answering (EQA) is traditionally evaluated under an episodic formulation, where agents solve each task independently and reset internal state between episodes.
However, real-world robots operate continuously and must accumulate, retain, and selectively reuse information acquired from prior interactions.
Despite this practical requirement, the architectural mechanisms needed to support sequential memory in EQA remain underexplored.
In this work, we investigate how different memory architectures behave when EQA agents are evaluated sequentially, with multiple questions answered in the same scene while memory is carried forward across queries.
We find that simply preserving existing memory is often insufficient

Sequential Evaluation Protocol

Traditional episodic evaluation treats each question as independent: the agent is reset and no internal memory, maps, or perceptual representations carry over. This simplifies evaluation but ignores whether agents can accumulate knowledge from previous interactions.

We instead evaluate sequences of questions in the same environment continuously without resetting internal state, exposing whether memory persistence actually improves later answers.

Sequential Evaluation Protocol — Beyond Episodic Evaluation: Memory Architectural Bottlenecks in Sequential Embodied Question Answering — (a) 3D-Mem trajectories on a representative MP3D scene. Under episodic evaluation, the agent re-explores from scratch for each query, answering only 1/3 correctly. Persistent 3D memory enables more direct navigation and correct answers for all 3 questions.

Memory Representation Paradigms in EQA

Existing EQA systems differ in how they connect perception, memory, and action. VLM-agent methods keep a pretrained VLM frozen and collect structured observations for inference, using memories such as 2D maps [ 16 ] , dense semantic libraries [ 26 ] , or 3D reconstructions [ 23 ] . This makes memory design modular, but also means that retrieval quality and representation structure strongly determine performance.

VLA methods instead fine-tune the VLM to output actions or waypoints end-to-end, with memory implicit in short-horizon hidden states.

Memory Representation Paradigms in EQA — Beyond Episodic Evaluation: Memory Architectural Bottlenecks in Sequential Embodied Question Answering — (b) UniNaviD trajectories on another MP3D scene. The agent answers 2/3 questions correctly episodically; sequential state inheritance corrects Q2, yielding 3/3. Unlike 3D-Mem, navigation efficiency changes minimally between settings.

Experimental Results and Analysis

Dataset. We construct our sequential benchmark by re-grouping questions from the OpenEQA [ 14 ] dataset by scene and evaluate the aforementioned 4 different types of representative agents. OpenEQA provides real indoor environments paired with free-form questions that require spatial and semantic reasoning.

Under our sequential protocol, all selected questions belonging to the same scene are concatenated into one ordered sequence, and the agent’s internal state is preserved across them without any resets.

Experimental Results and Analysis — Beyond Episodic Evaluation: Memory Architectural Bottlenecks in Sequential Embodied Question Answering — (a) Question 1: Indoor (”What color is the pot next to the black chair?”) and Outdoor (”What’s the color of the cone?”)

Real-Robot Validation

To verify that our simulated findings reflect physical deployment conditions, we evaluate all four methods on a real mobile robot in diverse environments. These experiments test whether the simulated bottlenecks — distributional shift under sequential context and lack of structured spatial memory — persist under sensing noise, actuation error, and environmental variability.

We deploy a Unitree Go2 quadruped with an Intel RealSense D435i depth camera and onboard LiDAR L2.

Real-Robot Validation — Beyond Episodic Evaluation: Memory Architectural Bottlenecks in Sequential Embodied Question Answering — (b) Question 2: Indoor (”How many yellow chairs are in this room?”) and Outdoor (”What’s the object behind the cone?”)

Sources and Demonstrations

Figures are reproduced from Cai et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.