Actively Resolving Contextual Uncertainty for Underspecified Tasks in Natural Language

Zachary Ravichandran, Jonathan Diller, Fernando Cladera, Varun Murali, George J. Pappas, Vijay Kumar · N/A · 2026

Summary

Abstract Summary

Foundation models provide robots with the ability to interpret natural language and reason about environmental context, yet most language-conditioned policies assume that goals are well-specified and that task-relevant information is provided upfront via a prior map. Operating in unfamiliar environments with underspecified tasks entails high contextual uncertainty: the robot must jointly infer what constitutes task success, what constitutes relevant information, and where (or whether) that information exists. We address these limitations via CLUE (Closed-Loop contextual Uncertainty rEsolution), a framework for actively resolving contextual uncertainty given underspecified tasks in natural language. CLUE uses an LLM-derived policy to hypothesize task-relevant concepts and potential plans. It then uses a language-embedded map, which is constructed online, to ground these hypotheses into actions. The policy sequentially evaluates hypotheses via closed-loop environment interaction and refines its plans as it gathers new information. We deploy CLUE on a Boston Dynamics Spot across three real indoor and outdoor environments spanning 15 tasks that require object disambiguation, functional inference, and occlusion reasoning. CLUE achieves a success rate within 7 percentage points of an oracle policy and outperforms an LLM-enabled planner without closed-loop feedback by a 4x margin. Supporting experiments demonstrate that simply building and then querying a language-enriched map is insufficient to resolve complex contextual planning tasks; these approaches achieve roughly one third the success rate of CLUE while requiring over 10x more VLM tokens. We provide additional information at https://zacravichandran.github.io/CLUE.

Key Points

Foundation models provide robots with the ability to interpret natural language and reason about environmental context, yet most language-conditioned policies assume that goals are well-specified and that task-relevant information is provided upfront via a prior map.
Operating in unfamiliar environments with underspecified tasks entails high contextual uncertainty: the robot must jointly infer what constitutes task success, what constitutes relevant information, and where (or whether) that information exists.
We address these limitations via CLUE (Closed-Loop contextual Uncertainty rEsolution), a framework for actively resolving contextual uncertainty given underspecified tasks in natural language.
CLUE uses an LLM-derived policy to hypothesize task-relevant concepts and potential plans
It then uses a language-embedded map, which is constructed online, to ground these hypotheses into actions.

Problem Definition

A robot is placed in an unknown environment and must achieve an underspecified task $\ell$ in natural language. The robot is equipped with a set of high-level behaviors for navigation, inspection, and manipulation, an odometry system, and an RGB-D camera.

Actively Resolving Contextual Uncertainty with CLUE

We address the above problem via CLUE , a closed-loop framework that pairs an LLM-enabled policy and a language-embedded mapping framework, as shown in Figure 2 . The policy has access to two categories of actions: hypothesis grounding and active information gathering . In the first, the policy grounds hypotheses, which include task-relevant entities, into concrete action objectives.

In the second, the policy actively validates hypotheses via inspection, navigation, and manipulation, or otherwise makes progress towards the task.

Actively Resolving Contextual Uncertainty with CLUE — Actively Resolving Contextual Uncertainty for Underspecified Tasks in Natural Language — Figure 3 : Hypothesis grounding. Sensory data is used to construct a language-embedded map. Given a concept from the policy’s hypothesis, candidate entities are identified by finding high-relevancy voxels and clustering. Viewpoints from which the candidates may be observed are identified via the map’s occupancy grid. These candidates and viewpoints form a graph which is provided to the policy via a textual observation along with a minimum distance visitation order.

Experimental Evaluation

We design experiments to evaluate the contributions claimed in § 1 . Concretely, we aim to answer the following questions: Q1. Can CLUE realize tasks that require resolving contextual uncertainty? Q2. Is closed-loop feedback required, or does open-loop planning suffice? Q3. Is CLUE ’s grounding process necessary, or does simply building and then querying a semantic map suffice?

We describe our implementation details in § 5.1 , our experimental setup—including environments, tasks, and baselines—in § 5.2 .

Experimental Evaluation — Actively Resolving Contextual Uncertainty for Underspecified Tasks in Natural Language — Figure 4 : Experimental snapshots showing the evaluation environments: two indoor settings and one outdoor setting.

Sources and Demonstrations

Figures are reproduced from Ravichandran et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.