LA4VLA: Learning to Act without Seeing via Language-Action Pretraining

Tao Lin, Yuxin Du, Yiran Mao, Zewei Ye, Yilei Zhong, Bing Cheng, Yiming Wang, Jiting Liu, Yang Tian, Junchi Yan, Feiran Wu, Zenan Meng, Hu Wei, Yuqian Fu, Gen Li, Bo Zhao · · 2026

Summary

Abstract Summary

Vision-Language-Action (VLA) models are commonly pretrained on robot demonstrations by jointly mapping visual observations and language instructions to actions. However, dense visual-action supervision can dominate the comparatively sparse language-action signal. As a result, policies may rely on visual shortcuts rather than learn how language conditions action execution, making them sensitive to visual variations. To address this limitation, we propose LA4VLA, a language-action pretraining framework that enables policies to acquire language-conditioned action priors without visual observations. These priors capture reusable manipulation skills shared across tasks and scenes, reducing reliance on scene-specific visual cues. Specifically, LA4VLA decomposes expert demonstration trajectories into atomic action segments and pairs each segment with a corresponding low-level action description. This yields LA4-33K, a dataset of 33K Language-Action (LA) episodes derived entirely from existing demonstrations without additional robot data collection. We further develop LA4VLA-1B, a lightweight 1B-parameter VLA model, and investigate three paradigms for incorporating language-action supervision into VLA learning: LA-only pretraining, sequential LA-to-VLA pretraining, and mixed LA-VLA pretraining. Across simulation and real-world tasks, LA-pretrained policies consistently outperform matched VLA-pretrained counterparts, while combining LA and VLA supervision leads to further gains. In particular, mixed LA-VLA pretraining improves the average success rate of LA4VLA-1B over the no-pretraining baseline by up to 17.8 and 45.0 percentage points in simulation and real-world tasks, respectively. These results establish LA4VLA as an effective and complementary pretraining strategy for building stronger and more robust VLA policies.

Key Points

Vision-Language-Action (VLA) models are commonly pretrained on robot demonstrations by jointly mapping visual observations and language instructions to actions.

Probing Instruction Following in VLAs

To examine whether a VLA policy predicts actions by following the instruction or by exploiting the visual observations paired with that instruction, we design a controlled diagnostic study.

Taking the standard paired VLA input as the reference, we keep the language instruction fixed while removing, mismatching, or conflicting the visual input, and then systematically examine how the predicted action trajectory changes.

Probing Instruction Following in VLAs — LA4VLA: Learning to Act without Seeing via Language-Action Pretraining — Figure 2 : Qualitative visualization of instruction-conditioned direction following under different visual input configurations. The top row shows predicted trajectories projected onto the plane containing the tested motion axis, and the bottom row presents endpoint displacement projected onto that axis.

LA4VLA: Language-Action Dataset Construction

The diagnostic analysis above shows that the instruction-following behavior of VLA policies can become unstable under changes in visual input. This suggests that language-action supervision should be exposed more directly, rather than always being tied to paired visual observations.

To this end, we propose the LA4VLA framework for vision-agnostic language-action pretraining, where models learn from language instructions, proprioceptive states, and action trajectories without visual inputs.

LA4VLA: Language-Action Dataset Construction — LA4VLA: Learning to Act without Seeing via Language-Action Pretraining — Figure 3 : Overview of the Language-Action dataset. The left panel summarizes the construction pipeline, which converts standard vision-language-action demonstrations into vision-agnostic atomic language-action episodes. The top-right panel visualizes representative Language-Action data examples using end-effector trajectories and gripper-direction arrows. The bottom-right panel reports the action-category distribution of the LA4-33K dataset.

LA4VLA: Language-Action Pretraining

This section systematically evaluates language-action pretraining as a VLA learning paradigm, examining its downstream performance, integration with VLA pretraining, transferability, robustness, and effects on instruction-conditioned action prediction. Pretraining paradigms. We study a pretraining paradigm built around vision-agnostic language-action supervision for VLA policy learning.

We use two paired pretraining data formats derived from the same atomic episodes.

LA4VLA: Language-Action Pretraining — LA4VLA: Learning to Act without Seeing via Language-Action Pretraining — Figure 4 : Real-world task execution on the xArm6 platform. Each row shows a representative rollout of one language-conditioned task, with frames ordered from left to right over the course of execution: Press Button (top), Place Book (middle), and Place Drink (bottom).

LA Dataset Construction Details

This appendix complements the dataset construction pipeline described in Section 4 . The main text presents the overall flow from original VLA data to the final LA dataset. Here we provide the implementation details for keyframe detection, atomic-action design criteria, prompt construction, VLM-based temporal segmentation, primitive definitions, and human verification. Purpose.

Keyframes provide temporal cues that can assist the segmentation model in identifying candidate atomic-action phases.

LA Dataset Construction Details — LA4VLA: Learning to Act without Seeing via Language-Action Pretraining — Figure 5 : Direction-following behavior and representation analysis of the LA-pretrained policy. Left panel: qualitative direction-following under the LA-pretrained and Conflicting-state conditions for a representative pair of opposite commands along the $z$ axis. The top row shows predicted trajectories in the $dy$–$dz$ plane, and the bottom row shows endpoint projections along the tested $z$ axis. Right panel: t-SNE visualization of the policy representations before action decoding, colored by direction command. The VLA-trained policy maps different commands to mixed clusters, whereas the LA-pretrained policy forms clearer command-aligned clusters.

Real-World Experimental Setup

Platform. The real-world experiments use a UFactory xArm6 arm with a parallel-jaw gripper. The policy receives two RGB camera views, one from a wrist-mounted camera and one from a third-person camera. Tasks. We design three language-conditioned manipulation tasks. In Press Button , four buttons are placed at different positions on the table, and the policy must press the button specified by the instruction.

In Place Book , the policy must insert a book into one of three shelf positions specified by the instruction.

Metrics for Instruction Following Analysis

Notations. The evaluation set is denoted as where $y_{i}\in\mathcal{Y}$ is the instruction direction label, $\hat{\mathbf{p}}_{i}\in\mathbb{R}^{3}$ is the predicted endpoint displacement, $\mathbf{g}_{i}\in\mathbb{R}^{3}$ is the ground-truth endpoint displacement under the corresponding instruction, and $\mathbf{u}_{i}\in\mathbb{R}^{3}$ is the unit vector corresponding to the instruction-specified canonical direction.

For example, $\mathbf{u}_{i}=(1,0,0)$ for $x^{+}$ and $\mathbf{u}_{i}=(-1,0,0)$ for $x^{-}$.

Sources and Demonstrations

Figures are reproduced from Lin et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.