OVMAN: A Task and Benchmark for Open-Vocabulary Motion-Aware Navigation

Summary

Abstract Summary

Homes change between a robot's visits. Navigation benchmarks pose their goals in the world the agent currently sees, and the two-visit benchmarks that exist score recall or rearrangement rather than navigation. None of them can express go to the chair that was moved or go to where the vase used to be. OVMAN is a task in which an agent tours a scene, returns after a scripted change, and must navigate to a goal specified by the change itself. Two of its six change relations answer with a place an object has left, where nothing remains to be detected. We release 219 two-visit episodes, each certified solvable by an oracle agent that completes it three times. Two released systems fail as predicted. A zero-shot object-goal navigator reaches the change-defined target in 11.4% of episodes and essentially fails at vacated locations, 0.000 on former and 0.025 on removed. A self-maintaining open-vocabulary map answers past-tense queries at about a third of the rate of the same maps read as two visits, even when grounding is supplied. A simple two-visit reference agent reaches 45.2% when navigating, against an embodied oracle of 99.5%. An error decomposition places the remaining difficulty in open-vocabulary instance grounding rather than in geometry or in selecting the answer once positions are known.

Key Points

Homes change between a robot's visits
Navigation benchmarks pose their goals in the world the agent currently sees, and the two-visit benchmarks that exist score recall or rearrangement rather than navigation.
None of them can express go to the chair that was moved or go to where the vase used to be
OVMAN is a task in which an agent tours a scene, returns after a scripted change, and must navigate to a goal specified by the change itself.
Two of its six change relations answer with a place an object has left, where nothing remains to be detected.

Methodology

Let $\mathcal{K}_{v}$ be the set of object instances present in visit $v$, and write the world state as $\mathcal{W}_{v}=\{(k,c_{k},x^{(v)}_{k}):k\in\mathcal{K}_{v}\}$ with identity $k$, category $c_{k}$ and floor-plane position $x^{(v)}_{k}\in\mathbb{R}^{2}$.

An OVMAN episode is a triple $(\mathcal{S},\delta,q)$: a scene $\mathcal{S}$ in state $\mathcal{W}_{1}$, a scripted change $\delta$ that produces $\mathcal{W}_{2}=\delta(\mathcal{W}_{1})$ while the agent is absent, and a query $q=(c,\rho)$ naming a category and a change relation.

Methodology — OVMAN: A Task and Benchmark for Open-Vocabulary Motion-Aware Navigation — Fig. 1: The six relations on recorded episodes from the released set, one column each: the agent’s view on the familiarisation tour (top) and from the same tour after the change (bottom), with the sentence it receives. The pin marks the answer, a floor-plane position: a filled star where the answer is an object that must be visible at the stop, a hollow star where the answer is a place the object has left. Frames are selected automatically by projecting the answer into the recorded views, none by hand.

Sources and Demonstrations

Figures are reproduced from Ghosh See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.