When Listening Becomes Easier: Scrubbing Visual Cues for Shortcut-Free VLAs

Jasper Gerigk, Kenzo Aspuru-Takata, Chin-Hsuan Wu, Mohammad Mohammadi, Shuhong Zheng, Igor Gilitschenski · N/A · 2026

Summary

Abstract Summary

Shortcut learning is a prevalent issue in robot learning. The limited diversity of robot demonstration datasets can mislead policies into exploiting spurious correlations between tasks and irrelevant features, such as viewpoint or background. Collecting sufficiently diverse robot demonstrations is costly and inefficient, motivating algorithmic alternatives. We focus on vision-language-action (VLA) models and discover that different vision-language model backbones exhibit substantially different levels of susceptibility to visual shortcut learning. We find that model behavior correlates with our proposed representation-level metric, action margin, which requires no policy rollouts. Visual shortcuts consistently enter action representations in early layers, with models differing in the extent to which later layers correct them by incorporating language information. To boost models' attention to language, we introduce task scrubbing, a new domain-adversarial training method that decreases models' likelihood of using visual shortcuts and improves VLAs' generalization. Experiments in both simulation and the real world across multiple VLAs and visual cues show that task scrubbing improves out-of-distribution robustness and often eliminates visual shortcut learning.

Key Points

Shortcut learning is a prevalent issue in robot learning
The limited diversity of robot demonstration datasets can mislead policies into exploiting spurious correlations between tasks and irrelevant features, such as viewpoint or background.
Collecting sufficiently diverse robot demonstrations is costly and inefficient, motivating algorithmic alternatives.
We focus on vision-language-action (VLA) models and discover that different vision-language model backbones exhibit substantially different levels of susceptibility to visual shortcut learning.
We find that model behavior correlates with our proposed representation-level metric, action margin, which requires no policy rollouts.

Problem Formulation and Analysis

We first establish a formal definition of visual shortcut learning. Let $T$ denote the task specified by language, $U$ denote the task-relevant visual information needed to execute the task, $Z$ denote a task-irrelevant visual cue, and $A$ denote the desired action. The complete visual observation is $X=(U,Z)$.

In the training distribution, task and cue identity are statistically dependent such that $I_{\mathrm{train}}(T;Z)>0.$ In our case, shortcut learning refers to cases where the learned policy mistakenly relies on $Z$ rather than $T$.

Problem Formulation and Analysis — When Listening Becomes Easier: Scrubbing Visual Cues for Shortcut-Free VLAs — Fig. 1: Visual shortcut-learning setup. During training (left), the model sees two different tasks with visual cues that match the language instruction. When the model is then evaluated (right) out-of-distribution where the visual cue does not match the language, the baseline model follows the visual cue rather than the language. Applying task scrubbing during training encourages the policy to follow the instruction.

Task Scrubbing

Task scrubbing is a training-time intervention that makes visually encoded task identity less stable and thus less convenient for the model to exploit during policy training. As summarized in Figure 5 , it attaches an adversarial task classifier to the visual pathway: the classifier learns to recover task identity, while the visual pathway learns to frustrate that classifier.

Because the resulting visual representations continually change, they provide the policy with a less stable signal for task identity.

Task Scrubbing — When Listening Becomes Easier: Scrubbing Visual Cues for Shortcut-Free VLAs — Fig. 2: OOD success and SC across VLA variants and low, medium, and high task-correlated viewpoint disparity on LIBERO Spatial.

Results

We evaluate task scrubbing on three separate experimental setups ( Figure 6 ), involving three types of visual shortcut. Returning to the setup from Section III-B , we evaluate task scrubbing on the high-disparity LIBERO [ 22 ] setting, where the two viewpoints differ by the largest rotation and the visual shortcut is the strongest.

We compare the baseline policy with task scrubbing and report mean and sample standard deviations across three seeds. Each seed uses 40 rollouts for each of the four task–viewpoint configurations.

Results — When Listening Becomes Easier: Scrubbing Visual Cues for Shortcut-Free VLAs — Fig. 3: Action margin shows a strong correlation with shortcut rate for the 21 model–disparity combinations. The representation-level metric strongly tracks shortcut reliance (Kendall’s $\tau_{b}=0.83$, $p=1.35\times 10^{-7}$).

Sources and Demonstrations

Figures are reproduced from Gerigk et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.