Decoding Task Progress from VLA Representations

Atiksh Bhardwaj, Edward Weiyi Duan, Prithwish Dan, Wei-Chiu Ma, Preston Culbertson · N/A · 2026

Summary

Abstract Summary

Vision-language-action models (VLAs) are moving rapidly towards deployment as general-purpose manipulation policies, but we currently lack basic tools for understanding what these models represent internally or for monitoring them at runtime. Leveraging ideas from mechanistic interpretability, we probe the residual stream of $π_{0.5}$ and find that task progress, the normalized time remaining in a trajectory, is linearly readable from the activations. We find that this signal is present in the pretrained PaliGemma backbone prior to training on any robot-specific data. A single linear probe generalizes to unseen tasks and varies under language counterfactuals when trained on multi-prompt data, but does not enable meaningful steering of the policy. These properties make the signal directly useful for instrumenting deployed VLAs. We use the probe as a simple label-free OOD detector, which detects stalled task progress, and find it competitive with state-of-the-art methods. Our results suggest that VLAs have rich, linearly readable internal representations of semantic quantities like task progress, and that learning to read these signals offers a lightweight, interpretable path toward monitoring deployed visuomotor policies.

Key Points

Vision-language-action models (VLAs) are moving rapidly towards deployment as general-purpose manipulation policies, but we currently lack basic tools for understanding what these models represent internally or for monitoring them at runtime.
Leveraging ideas from mechanistic interpretability, we probe the residual stream of $π_{0.5}$ and find that task progress, the normalized time remaining in a trajectory, is linearly readable from the activations.
We find that this signal is present in the pretrained PaliGemma backbone prior to training on any robot-specific data.
A single linear probe generalizes to unseen tasks and varies under language counterfactuals when trained on multi-prompt data, but does not enable meaningful steering of the policy.
These properties make the signal directly useful for instrumenting deployed VLAs

Methods

We define the transformer architecture commonly found in VLAs including our targeted VLA model of $\pi_{0.5}$ [ 10 ] and its vision-language backbone of PaliGemma [ 2 ] .

Given an input observation $\bm{x}=(\bm{I},\bm{q},\ell)$ which consists of an image $\bm{I}$, proprioceptive data $\bm{q}$, and a language prompt $\ell$, the VLA model processes this sequence through multiple layers of transformer blocks, producing a sequence of intermediate embeddings at each layer.

Methods — Decoding Task Progress from VLA Representations — Figure 2: Left: Weak decodability of the progress feature across layers of fine-tuned $\pi_{0.5}$ (SigLIP, PaliGemma’s Gemma, and the action expert). Right: Probe performance vs. capacity on the first Gemma layer, true labels versus shuffled labels

Experiments and Results

Our experiments evaluate the progress feature along four axes: weak decodability, strong decodability, steerability, and OOD detection. The target model is $\pi_{0.5}$, fine-tuned on VLABench [ 30 ] , which provides 10 manipulation tasks with semantic variation and 500 trajectories per task. We follow the procedure from Subsection 3.1 to collect the embeddings and labels for $\tau$.

Q1: Can we weakly decode the progress feature across the layers of the VLA model? We train one linear probe per layer of $\pi_{0.5}$ under an $L_{1}$ loss (Fig.

Experiments and Results — Decoding Task Progress from VLA Representations — Figure 3: Weak decodability of the progress feature across four models, probed at successive stages of training: PaliGemma (vision-language backbone only), base $\pi_{0.5}$ (after large-scale robot pre-training, before fine-tuning), and VLABench fine-tuned $\pi_{0}$ (a distinct VLA architecture), and VLABench fine-tuned $\pi_{0.5}$ (our target model).

Sources and Demonstrations

Figures are reproduced from Bhardwaj et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.