Self-Adaptive VLA for Robust Robot Deployment

Hongxin Zhang, Chunru Lin, Tsun-Hsuan Wang, Zhenjia Xu, Chuang Gan · N/A · 2026

Summary

Abstract Summary

While Vision-Language-Action (VLA) models demonstrate impressive capabilities in robotic manipulation, their memoryless nature renders them brittle to test-time environment shifts, particularly hardware shifts caused by wear or imperfect calibration. Enabling these models to self-adapt during deployment without requiring continuous on-site recalibration remains a critical bottleneck for real-world scalability. In this work, we introduce Self-Adaptive VLA, a novel post-training recipe that enables the policy to iteratively adapt to deployment-time hardware shifts leveraging its own rollouts as context. To do so, we first collect policy rollouts under deliberately injected hardware shifts. We then transform the base policy's training data into shift-conditioned expert demonstrations by pre-compensating the expert actions for these known shifts. Next, we introduce a lightweight, plug-in context encoder that compresses the context, including visual observation, proprioception, and actions in the shifted environment, into a latent context token. This token modulates the policy through adaptive layer normalization (AdaLN). Furthermore, we find that context tokens can be ensembled, allowing the policy to iteratively self-correct and mitigate failures step by step. Extensive experiments across four precision-critical bi-manual and dexterous manipulation tasks show that Self-Adaptive VLA recovers over 80% of the base policy's performance under hardware shifts, such as actuation bias and joint encoder offsets. Moreover, Self-Adaptive VLA enables more robust deployment to new workstations compared to the base policy. Our approach provides a pathway for robust large-scale real-world robot deployments and easier maintenance. See videos at https://icefoxzhx.github.io/self-adaptive-vla.

Key Points

While Vision-Language-Action (VLA) models demonstrate impressive capabilities in robotic manipulation, their memoryless nature renders them brittle to test-time environment shifts, particularly hardware shifts caused by wear or imperfect calibration.
Enabling these models to self-adapt during deployment without requiring continuous on-site recalibration remains a critical bottleneck for real-world scalability.
In this work, we introduce Self-Adaptive VLA, a novel post-training recipe that enables the policy to iteratively adapt to deployment-time hardware shifts leveraging its own rollouts as context.
To do so, we first collect policy rollouts under deliberately injected hardware shifts
We then transform the base policy's training data into shift-conditioned expert demonstrations by pre-compensating the expert actions for these known shifts.

Problem Setup

We study Vision-Language-Action (VLA) policies $\pi(A_{t}/l,o_{t},q_{t})$ that learns to predict a chunk of $h$ future actions $A_{t}=a_{t+1,...,t+h}$ given language instruction $l$, an image observation $o_{t}$ and a proprioceptive state $q_{t}$. The base policies are learned or fine-tuned from human expert demonstrations $\mathcal{D}_{base}$ collected under a specific training hardware configuration $H_{train}$.

Method

We propose a novel post-training recipe Self-Adaptive VLA to enable VLAs to leverage failure rollouts as context for adapting to test-time hardware shifts. In Section IV-A , we introduce an automated pipeline to synthesize hardware-shift-conditioned expert data from standard demonstration datasets.

We then detail the design of the plug-in context encoder and its AdaLN-based integration with the base VLA policy in Section IV-B . Finally, Section IV-C outlines our ensembling strategy for iterative, step-by-step policy adaptation.

Method — Self-Adaptive VLA for Robust Robot Deployment — Fig. 4 : Task Overview . We show the initial and goal frames along with key frames for the four evaluated tasks. Each frame is composed of the ego camera view and two wrist views on top. Tasks range $10\sim 15$ seconds and require precise grasping and insertion.

Experiments

We conduct experiments on four bi-manual and dexterous manipulation tasks requiring precise grasping and insertion (overview in Figure 4 ). Three tasks, Transport Corn , Insert Tube , and Cap Marker , are executed using 6-DoF Piper or Piper-X arms equipped with parallel grippers. The fourth task, Assemble Ring , utilizes two 7-DoF Marvin Arms paired with a 20-DoF Wuji Hand.

All robots are equipped with one ego camera and two wrist cameras. For each task, we first train a base policy using human-teleoperated expert demonstrations.

Experiments — Self-Adaptive VLA for Robust Robot Deployment — Fig. 5 : Qualitative results. We visualize key frames across the timeline for the Cap Marker and Insert Tube tasks. The boundary between the green and red timeline segments denotes the first visually identifiable failure by a human.

Sources and Demonstrations

Figures are reproduced from Zhang et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.