What Matters, When? Diagnosing and Improving Conditional Visual Grounding in Visuomotor Imitation Policies

Vivek Chavan, Pengtao Xie, Yahuan Shi, Oliver Heimann, Kevin Haninger, Jörg Krüger · N/A · 2026

Summary

Visuomotor imitation policies can achieve high performance under in-distribution visual conditions yet fail when visually similar objects or receptacles are introduced.

Abstract Summary

Visuomotor imitation policies can achieve high performance under in-distribution visual conditions yet fail when visually similar objects or receptacles are introduced. We study this behavior as a problem of conditional visual grounding: the visual target required for successful control changes with the manipulation phase and, in more complex tasks, with the observed task state. Using Action Chunking with Transformers (ACT), we systematically introduce distractor objects and receptacles with controlled color and shape similarity and localize failures to picking and placement. We find that distractor sensitivity is specific to both the type of visual similarity and the manipulation stage. Guided by this diagnosis, we evaluate distractor augmentation, phase-dependent attention regularization, and appearance-based visual prompting as complementary interventions for improving target selection while preserving spatial information required for control. These interventions substantially improve robustness in simulation and on a physical UR3e. We further examine the same failure pattern in a pretrained vision-language-action policy on a state-conditioned instrument-handling task, where the observed state of a medical instrument determines the correct destination. Together, the results show that visual distractors can cause incorrect object or destination selection even when the underlying manipulation skill remains intact, and that explicitly improving target selection can substantially recover performance across distinct visuomotor policy-learning regimes.

Key Points

Visuomotor imitation policies can achieve high performance under in-distribution visual conditions yet fail when visually similar objects or receptacles are introduced.
We study this behavior as a problem of conditional visual grounding: the visual target required for successful control changes with the manipulation phase and, in more complex tasks, with the observed task state.
Using Action Chunking with Transformers (ACT), we systematically introduce distractor objects and receptacles with controlled color and shape similarity and localize failures to picking and placement.
We find that distractor sensitivity is specific to both the type of visual similarity and the manipulation stage.
Guided by this diagnosis, we evaluate distractor augmentation, phase-dependent attention regularization, and appearance-based visual prompting as complementary interventions for improving target selection while preserving spatial information required for control.

Method

We study two simulated pick-and-place tasks designed to separate target and destination grounding. In Task 1, a randomly placed object is moved to a fixed receptacle; in Task 2, both object and receptacle positions are randomized, requiring visual localization during both picking and placement. ACT is trained from 100 clean scripted demonstrations per task.

At evaluation, held-out object and receptacle competitors match the target in color, shape, or neither, with one to three competitors per class.

Method — What Matters, When? Diagnosing and Improving Conditional Visual Grounding in Visuomotor Imitation Policies — Fig. 1: ACT-Modified architecture. The full camera observation and phase-selected visual prompt are encoded into the visual memory consumed by the ACT transformer decoder. A phase predictor selects the object or receptacle prompt according to the current manipulation phase, while robot proprioception ( qpos ) provides state conditioning. The decoder combines visual memory, learned queries, cross-attention, and positional embeddings to predict an action sequence. The action-sequence encoder and CLS latent path form the CVAE posterior during training only; during inference, future actions are unavailable and the latent is fixed to the prior mean $z=0$.

Experiments and Results

Standard ACT reaches 98.5% and 99.7% success in clean Tasks 1 and 2, confirming that both manipulation routines are learned. Under three color-matched object competitors in Task 2, $P(\mathrm{pick})$ falls to 39.2%, while $P(\mathrm{lift}\mid\mathrm{pick})$ and $P(\mathrm{place}\mid\mathrm{pick},\mathrm{lift})$ remain 93.8% and 95.5%.

In contrast, two shape-matched receptacle competitors leave $P(\mathrm{pick})$ at 97.5% but reduce conditional placement to 33.9%.

Experiments and Results — What Matters, When? Diagnosing and Improving Conditional Visual Grounding in Visuomotor Imitation Policies — Fig. 2: Robust placement requires both distractor invariance and retained target geometry. The top row compares clean and full-distractor Task 2 place-phase decoder states. The bottom row colors the corresponding representations by the target container’s left/right position, with clean and full conditions shown in a shared within-model projection. Augmentation-only ACT exhibits the strongest clean/full invariance, whereas ACT-Modified preserves substantially clearer container-position structure. The t-SNE projections are illustrative; quantitative comparisons in Table II are computed in the original feature space.

Sources and Demonstrations

Figures are reproduced from Chavan et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.