MIND: Instruction-Conditioned Incremental Grounding for Robots

Summary

Robots following persistent language instructions must decide which cached room judgments remain valid as their environment changes.

Abstract Summary

Robots following persistent language instructions must decide which cached room judgments remain valid as their environment changes. We present MIND (Maintenance via INstruction-conditioned Dependencies), a framework for maintaining target-room decisions without re-evaluating every room after each event. A language model compiles each instruction into dependencies for semantic qualification and target selection. The controller intersects semantic dependencies with event footprints, refreshes affected judgments, and updates reachability and selection separately. For fixed deterministic evaluation, planning, and selection, conservative dependency and footprint coverage preserves the decisions of full recomputation. On 240 five-event ProcTHOR sequences with an exact benchmark predicate, MIND reduces mean semantic evaluations from 30.6 to 11.2 while preserving all 1,200 post-event decisions. Frozen-evaluator experiments with Phi-4 and Qwen3-14B also preserve all 1,200 decisions, without improving the evaluators' underlying task accuracy. In 32 closed-loop AI2-THOR episodes using exact-predicate qualification and metadata-derived footprints, MIND matches the final target decision of full recomputation in every episode. Unitree G1 experiments demonstrate target reuse and reselection across room transitions without additional semantic evaluations under MIND, and abstention after a barrier and an object removal at 4 evaluations against 15 to 18. Additional materials are available at https://mind-icra27.github.io/.

Key Points

Robots following persistent language instructions must decide which cached room judgments remain valid as their environment changes.
We present MIND (Maintenance via INstruction-conditioned Dependencies), a framework for maintaining target-room decisions without re-evaluating every room after each event.
A language model compiles each instruction into dependencies for semantic qualification and target selection.
The controller intersects semantic dependencies with event footprints, refreshes affected judgments, and updates reachability and selection separately.
For fixed deterministic evaluation, planning, and selection, conservative dependency and footprint coverage preserves the decisions of full recomputation.

Problem Formulation

System and event stream. An episode fixes a natural-language instruction $q$ and a building with regions $\mathcal{R}$; in our indoor experiments each region is one room. The episode then evolves through events $e_{1:T}$. Event $e_{t}$ changes the world from $X_{t-1}$ to $X_{t}$, where $X_{t}$ holds region inventories, semantic region labels, the robot’s region, interaction history, and the region-adjacency graph.

We denote these state categories by We instantiate $\mathcal{V}$ with five state categories common to indoor grounding.

Problem Formulation — MIND: Instruction-Conditioned Incremental Grounding for Robots — Fig. 1: An overview of “Bring me the nearest mug.” At $t_{0}$ every room is judged and cached. Moving ($t_{1}$) changes no room: judgments are reused, only the target is re-chosen. Removing the kitchen mug ($t_{2}$) refreshes one judgment. Full recomputation rejudges all four each time.

Methodology

MIND compiles dependencies once per instruction and uses event footprints to maintain cached room judgments, reachability, and target selection (see Fig. 2 ). The compiler (Fig. 2 ) maps $q$ to The true read sets are defined counterfactually for the fixed semantic function $s_{q}$ and selector $\pi_{q}$ used in the comparison.

A variable $v$ belongs to $D^{*}_{\mathrm{sem}}(q)$ if changing only state category $v$ can change $s_{q}(r;X)$ for some candidate region.

Methodology — MIND: Instruction-Conditioned Incremental Grounding for Robots — Fig. 2: Architecture for “bring me the nearest mug.” (A) Once per instruction, the compiler separates variables needed to judge each room from those needed only to choose among qualifying rooms. (B) After each event, the detector reports what changed and which rooms may be affected. (C) Their intersection fixes which room judgments to recheck; position and topology changes also update reachability. (D) The controller reuses untouched judgments and reselects a reachable target whenever the feasible set is recomputed or a reported change intersects either compiled dependency slot.

Benchmarks and Experimental Setup

The six-family dependency and request benchmarks use automatically checked manifests frozen by hash before model evaluation.

A dependency manifest is refused if a sequence has no verdict-changing event or abstains throughout, a history sequence is not moved by its use event, a move or door event changes a verdict for a nearest request, or no event exposes the inventory-only rule; a request manifest is refused if families are unbalanced or share a signature, a minimal pair adds an edge other than the one claimed, a held-out template leaks into d...

Benchmarks and Experimental Setup — MIND: Instruction-Conditioned Incremental Grounding for Robots — Fig. 3: Removal localization in ProcTHOR validation house 571. (A, B) Matched views before and after mug removal. (C) The candidate footprint contains the affected bedroom, the only room refreshed here. (D) Reselection identifies the kitchen mug; the living room remains unreachable.

Results

Every number below is measured on the frozen evaluation units of Section V with the metrics defined there. Semantic-evaluation counts are reported per five-event sequence. Table I reports episode-accuracy proportions with $95\%$ Wilson intervals and semantic-evaluation means with standard deviations.

Decision-agreement rates are descriptive proportions over the evaluated decisions; the observation-derived footprint experiment reports a one-sided Clopper–Pearson bound. Conservative coverage and update scope.

Results — MIND: Instruction-Conditioned Incremental Grounding for Robots — Fig. 4: Unitree G1 hardware validation. Left: the robot, with the 3D lidar and the RGB-D camera used for localization and detection. Centre: ceiling view of the four-zone laboratory, chairs in A and C and a backpack in the start zone D, with the path taken in one run of scenario 2, the nearest thing to sit on . Insets: the robot’s camera in each zone; the three moments of the run carry the target and the evaluation count from the module log. Three initial evaluations select C, one hop away, and the robot walks there and arrives. When the operator then drives it into A, the target re-ranks to A on position alone; position is a selection dependency for this request, so no room is re-judged.

Sources and Demonstrations

Figures are reproduced from Abouagour et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.