Retrospective Open-Vocabulary Memory for Long-Term Object Search

Summary

Long-term object search requires learning where objects usually appear from repeated but uneven observations of a changing environment.

Abstract Summary

Long-term object search requires learning where objects usually appear from repeated but uneven observations of a changing environment. We formulate retrospective open-vocabulary memory as probabilistic inference from censored observations, where the key idea is to reason with evidence per opportunity: a detection or non-detection should influence belief only in proportion to the robot's opportunity to observe the corresponding location. We introduce ECROM, which uses this principle to estimate long-term prevalence for concepts specified only at query time and converts the resulting belief directly into an active-search prior. To evaluate this problem, we introduce a controlled long-term benchmark in ten HM3D homes that independently varies object placement and observation opportunity across repeated traversals. ECROM improves support-level AP on held-out queries by 4.5 points and search SPL by 4.2 points over the strongest competing memory in each metric. The benchmark, dataset, and code will be open-sourced.

Key Points

Long-term object search requires learning where objects usually appear from repeated but uneven observations of a changing environment.
We formulate retrospective open-vocabulary memory as probabilistic inference from censored observations, where the key idea is to reason with evidence per opportunity: a detection or non-detection should influence belief only in proportion to the robot's opportunity to observe the corresponding location.
We introduce ECROM, which uses this principle to estimate long-term prevalence for concepts specified only at query time and converts the resulting belief directly into an active-search prior.
To evaluate this problem, we introduce a controlled long-term benchmark in ten HM3D homes that independently varies object placement and observation opportunity across repeated traversals.
ECROM improves support-level AP on held-out queries by 4.5 points and search SPL by 4.2 points over the strongest competing memory in each metric.

Problem Setup

We record episodes (traversals) $e\in\{1,\ldots,E\}$, each containing a stream of RGB-D frames, odometry, and timestamps. Our goal is to build a query-independent memory $\mathcal{M}$ that can later infer where an arbitrarily specified object is likely to occur and use this prior to search efficiently, rather than exploring the environment from scratch.

We organize this history as a place–view atlas built on the CROSS keyframe graph ( Wang et al., 2026 ) .

Problem Setup — Retrospective Open-Vocabulary Memory for Long-Term Object Search — Figure 2: System overview. (1) During query-independent mapping, RGB-D traversals are organized into a place–view atlas with surface supports and per-episode observation opportunities $c_{je}$. (2) At query time, open-vocabulary detector responses are lifted to supports and combined with opportunity to infer the long-term prevalence posterior $p(\theta_{j}\mid\mathcal{H}_{j},q)$ (Section 4.3 ). (3) The resulting belief guides budgeted active search and is updated after each executed view (Eq. 8 ).

Evidence per Opportunity

(a) Place graph and surface supports $j$ from traversals (b) Opportunity $c_{je}$ of each traversal (c) $p(\theta_{j}\mid\mathcal{H}_{j},q)\propto p(\theta_{j})\,\times$ $\prod_{e}\bigl[1+\theta_{j}c_{je}(\Lambda_{je}-1)\bigr]$ (d) $u^{\star}=\operatorname*{argmax}_{u}$ $\sum_{j}b_{j}c_{j}(u)/(C_{G}(\mathrm{cur},u){+}d_{0})$ 1. Mapping (query independent, once) 2. Query time (any text $q$) 3.

LOOP-Bench: Long-Term Object Occurrence Prediction

We introduce LOOP-Bench ( L ong-Term O bject O ccurrence P rediction), a controlled benchmark that tests whether a robot can infer where an object usually occurs from repeated, uneven observations.

It contains ten HM3D-Semantics homes ( Ramakrishnan et al., 2021 ; Yadav et al., 2023 ) , where objects are repeatedly placed from hidden long-term distributions while routes and viewpoints are sampled independently, separating prevalence from observation opportunity (Figure 4 ).

LOOP-Bench: Long-Term Object Occurrence Prediction — Retrospective Open-Vocabulary Memory for Long-Term Object Search — Figure 5: Real-robot experiment. Left : Hello Robot Stretch 3 (image: Hello Robot). Right : the $\approx$500 m 2 office floor with the ten mapping traversals (grey) and, for one query, the search routes of Ecrom (green), detection count (blue) and PredictiveGraphs-style (magenta) from the start (yellow dot) to the target (red star). Scale bar 5 m.

Simulation Experiments

For each of the 50 LOOP-Bench queries, we build each method’s memory from the 20 mapping traversals of its house and evaluate its predictions on the shared surface supports.

We derive two ground truths from the hidden placement distribution $\pi$: a 15 cm footprint allocates each placement to supports, giving the occurrence probabilities $g_{j}$ used for ranking, and $\theta_{j}^{\ast}=\sum_{k}\pi_{k}m_{jk}$ is the probability that the object rests within 45 cm of support $j$ ($m_{jk}=1$ when placement $k$ does), the prevalence of Eq.

Real-Robot Experiment

We test whether the prevalence learned by Ecrom transfers to physical robot search. A Hello Robot Stretch 3 operates on an approximately $500\,\mathrm{m}^{2}$ office floor containing desks, tables, shelves, and other candidate supports (Figure 5 ). Each object is assigned a hidden distribution over supports.

Before each of ten mapping traversals, its location is resampled from this distribution; the robot records only RGB-D observations and odometry.

Sources and Demonstrations

Figures are reproduced from Wang et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.