LT-Mem: Volatility-Aware Spatio-Temporal Memory for Lifelong Scene Understanding

Summary

Long-term robot operation in evolving environments requires object-level understanding that persists across repeated revisits.

Abstract Summary

Long-term robot operation in evolving environments requires object-level understanding that persists across repeated revisits. Existing systems either overwrite history to maintain an up-to-date map or store semantic snapshots without consistent cross-session object identity, resulting in temporal amnesia: the systematic loss of object history that prevents answering queries such as "Where has the green chair been across all sessions?" We propose LT-Mem, a volatility-aware memory evolution framework that unifies spatially aligned instance-level 3D perception with volatility-conditioned temporal reasoning. First, a multi-session SLAM backbone provides spatially aligned per-object observations across sessions. Second, a reasoning layer governs how object memory evolves: deterministic evidence scoring preserves cross-session identity, and a volatility-aware policy selects among overwrite, hold, and multi-hypothesis actions based on each object's dynamics. Third, the resulting Tri-Memory structure (Live, Delta, Meta) preserves both current states and event histories, enabling longitudinal object-centric reasoning. We further introduce LT-VQA, a dataset and evaluation suite comprising multi-session recordings, persistent identity annotations, and temporal QA pairs. Experiments show that LT-Mem consistently outperforms baselines across all metrics while consuming an order of magnitude fewer tokens, and ablations confirm that gains are driven by the structured memory architecture rather than LLM capacity.

Key Points

Long-term robot operation in evolving environments requires object-level understanding that persists across repeated revisits.
Existing systems either overwrite history to maintain an up-to-date map or store semantic snapshots without consistent cross-session object identity, resulting in temporal amnesia: the systematic loss of object history that prevents answering queries such as "Where has the green chair been across all sessions?" We propose LT-Mem, a volatility-aware memory evolution framework that unifies spatially aligned instance-level 3D perception with volatility-conditioned temporal reasoning.
First, a multi-session SLAM backbone provides spatially aligned per-object observations across sessions.
Second, a reasoning layer governs how object memory evolves: deterministic evidence scoring preserves cross-session identity, and a volatility-aware policy selects among overwrite, hold, and multi-hypothesis actions based on each object's dynamics.
Third, the resulting Tri-Memory structure (Live, Delta, Meta) preserves both current states and event histories, enabling longitudinal object-centric reasoning.

Method

We consider an environment observed across $T$ sessions (i.e., independent mapping runs conducted at different times) $S_{1},\ldots,S_{T}$, where objects may move, appear, disappear, or be temporarily occluded.

For each object $q$ in session $S_{t}$, the perception layer produces a 3D observation where $\mathbf{c}_{t}^{q}\in\mathbb{R}^{3}$ is the centroid, $v_{t}^{q}$ denotes volume, and $\mathbf{f}_{t}^{q}$ is a visual embedding extracted from the segmented region.

Method — LT-Mem: Volatility-Aware Spatio-Temporal Memory for Lifelong Scene Understanding — Fig. 3 : Overview of the three LT-VQA environments and dataset. Each column shows a globally aligned 3D point-cloud map with multi-session trajectories overlaid (top) and representative RGB frames (bottom). (Left) Lab-S : a compact indoor room with 10 tracked objects. (Middle) Lab-L : a larger indoor space with more complex layout. (Right) Parking Lot : an outdoor scene for spatial statistics evaluation. Trajectory colors distinguish individual sessions; object configurations change across sessions due to relocation and temporary disappearance.

Experiments

We construct LT-VQA, a controlled multi-session dataset for evaluating long-term object-centric scene understanding. It jointly provides globally aligned multi-session geometry, persistent object-instance identity, event-level state transition annotations, and session-indexed temporal QA pairs.

While 3RScan [ 20 ] provides instance-level correspondence across rescans, it lacks event-level annotations and temporal QA supervision.

Experiments — LT-Mem: Volatility-Aware Spatio-Temporal Memory for Lifelong Scene Understanding — Fig. 4 : Tri-Memory example for the robot dog in Lab-S over 10 sessions. The color overlay encodes the evolving volatility score $V_{t}$ (red: static, blue: volatile). Two move events are recorded in Delta Memory with metric context; Live Memory reflects the final confirmed position at S10, Meta Memory classifies the object as semi-static.

Sources and Demonstrations

Figures are reproduced from Lee et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.