Post-Training, Finetuning, and Steering:A Field Guide for Curating Training Data

Maria Attarian, Yifan Ruan, Claas Voelcker, Chenyang Cao, Amy Zhang, Peter Stone, Nicholas Rhinehart, Igor Gilitschenski · N/A · 2026

Summary

While Vision-Language-Action models have enabled impressive cross-task transfer in zero-shot settings, \emph{adaptation} methods are often necessary to achieve consistent performance on novel tasks.

Abstract Summary

While Vision-Language-Action models have enabled impressive cross-task transfer in zero-shot settings, \emph{adaptation} methods are often necessary to achieve consistent performance on novel tasks. However, even though a wealth of methods for adaptation have been proposed, it is unclear how to choose the best method based on available trajectories. We close this gap by studying the impact of the adaptation method under a variety of common data scenarios: on- and off-policy data, offline vs online adaptation, and whether the data contains optimal or suboptimal trajectories. Our evaluation spans simulated tasks in the LIBERO suite and real tasks on a Franka platform. In the offline data regime, critic-based steering approaches outperform supervised finetuning when failure data is available. However, offline RL techniques for training critic functions fail to outperform simpler Monte-Carlo-based approaches. Finally, in the online setting, critic-based steering is significantly more sample efficient compared to policy-gradient-based finetuning methods, but has a lower performance ceiling than full finetuning.

Key Points

While Vision-Language-Action models have enabled impressive cross-task transfer in zero-shot settings, \emph{adaptation} methods are often necessary to achieve consistent performance on novel tasks.
However, even though a wealth of methods for adaptation have been proposed, it is unclear how to choose the best method based on available trajectories.
We close this gap by studying the impact of the adaptation method under a variety of common data scenarios: on- and off-policy data, offline vs online adaptation, and whether the data contains optimal or suboptimal trajectories.
Our evaluation spans simulated tasks in the LIBERO suite and real tasks on a Franka platform
In the offline data regime, critic-based steering approaches outperform supervised finetuning when failure data is available.

An Overview of Policy Adaptation

We will first establish policy adaptation formally, and then discuss different data regimes used by various adaptation methods. We use policy adaptation to refer to any technique that improves a pre-trained policy’s performance on novel tasks. For pretrained foundation models, such as VLAs, this is also called post-training .

Since foundation model training can involve several pre- and post-training stages, we adopt a less ambiguous term. Policy adaptation techniques share a common structure.

Evaluation of Adaptation Approaches in Simulation

We test the ability of four representative approaches to improve performance of a pre-trained VLA on tasks from the LIBERO benchmark [ 30 ] , which offers a variety of pick-and-place tasks and setups. However, not all methods can be meaningfully combined with every data regime.

Supervised finetuning only uses success data, and is therefore inapplicable in a mixed data regime, while steering-based methods can be executed in every data regime.

Evaluation of Adaptation Approaches in Simulation — Post-Training, Finetuning, and Steering:A Field Guide for Curating Training Data — Fig. 2 : Results of the offline on-policy data LIBERO evaluation: SFT and CBS produce similar results when given the same success-only data, and CBS benefits from having access to more failure data. Changes to critic objective showed little effect on performance in our experiments.

Evaluation of Adaptation Approaches in the Real World

Our real world experiments serve to corroborate our main finding which is the strong impact of the data distribution on performance. As all critic training methods performed similarly to each other, we refrain from retraining all critic variants on real data. We use a Franka arm configured following the DROID setup [ 25 ] . $\pi_{0.5}$ finetuned on DROID data serves as the base policy.

Sources and Demonstrations

Figures are reproduced from Attarian et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.