Evidence-Gated Task and Motion Planning with Vision-Language Models

Tsunehiko Tanaka, Matthew Stephenson, Alistair Macvicar, Edgar Simo-Serra · N/A · 2026

Summary

Abstract Summary

Robots executing long-horizon manipulation tasks from natural-language instructions must reason about both semantic task structure and geometric feasibility. However, under partial observability, the availability of goal-relevant objects may be uncertain. In such cases, approaches that combine Vision-Language Models (VLMs) with Task and Motion Planning (TAMP) may generate subgoals that rely on the VLM's prior knowledge without observational support, leading to execution failures or unintended outcomes. We propose Evidence Acquisition and Feasibility Gating (EAFG), a framework that acquires visual evidence through VLM-generated exploratory subgoals and TAMP-based execution. EAFG then applies a feasibility gate to decide whether to proceed with task planning, acquire further evidence, or halt. Our experiments show that, in cooking tasks with ambiguous object use, EAFG improves recipe completion by discovering task-relevant objects before planning. For instructions requiring an absent object, EAFG promotes appropriate halt decisions and reduces repeated attempts to manipulate that object.

Key Points

Robots executing long-horizon manipulation tasks from natural-language instructions must reason about both semantic task structure and geometric feasibility.
However, under partial observability, the availability of goal-relevant objects may be uncertain
In such cases, approaches that combine Vision-Language Models (VLMs) with Task and Motion Planning (TAMP) may generate subgoals that rely on the VLM's prior knowledge without observational support, leading to execution failures or unintended outcomes.
We propose Evidence Acquisition and Feasibility Gating (EAFG), a framework that acquires visual evidence through VLM-generated exploratory subgoals and TAMP-based execution.
EAFG then applies a feasibility gate to decide whether to proceed with task planning, acquire further evidence, or halt.

Evidence Acquisition and Feasibility Gating

We propose Evidence Acquisition and Feasibility Gating (EAFG), which uses a VLM to actively acquire observational evidence for verifying goal-relevant objects and environmental conditions before task planning. EAFG generates evidence-acquisition subgoals that resolve missing or uncertain information, and uses the acquired evidence to determine whether planning for the goal should proceed.

Experiments

We use a kitchen environment for making chicken soup. The task requires long-horizon planning over object search beyond the initially visible area, access to storage spaces, placement of ingredients and seasonings, and the use of water and a stove.

The environment consists of five movable objects, eight support surfaces such as counters, stove burners, a sink, and a pot, two storage spaces enclosed by doors or a drawer, and six articulated objects such as doors and knobs.

Experiments — Evidence-Gated Task and Motion Planning with Vision-Language Models — Figure 2: Kitchen environment used in the experiments. (a) Initial state of the kitchen. (b) Layout of objects in the kitchen.

Sources and Demonstrations

Figures are reproduced from Tanaka et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.