DriftOPD: Sequence-Level Reverse-KL Distillation for One-Step VLA Policies

Youngjun Jun, Kyumin Choi, Youngmin Kim, Seonghyun Jin, Sunwoo Park, Jangho Park, Jong Chul Ye · N/A · 2026

Summary

Vision-Language-Action (VLA) models increasingly rely on action experts that generate short action chunks under receding-horizon control.

Abstract Summary

Vision-Language-Action (VLA) models increasingly rely on action experts that generate short action chunks under receding-horizon control. While chunk-level training is convenient across robot embodiments, it optimizes local action likelihood without explicitly accounting for long-horizon task success. Sequence-level reinforcement learning can address this limitation, but typically requires policy rollouts and closed-loop interaction, which are costly for real-robot manipulation. We introduce DriftOPD, a teacher-free, rollout-free framework for sequence-level on-policy distillation of continuous VLA action experts. We show that the sequence-level reverse Kullback-Leibler (KL) divergence decomposes into a chunk-level reverse-KL term and a future-potential term that captures the long-horizon effect of the current action. DriftOPD optimizes these two terms using a one-step drifting objective and a Q-function critic learned from offline demonstrations, respectively, enabling sequence-level optimization with only offline data and one-step action generation. Across multiple VLA architectures in simulation and real-world manipulation, DriftOPD generally outperforms existing one-step distillation baselines while achieving task success performance comparable to multi-step teacher policies. These results demonstrate that long-horizon behavior can be effectively distilled into one-step VLA action experts without online interaction or a separate teacher.

Key Points

Vision-Language-Action (VLA) models increasingly rely on action experts that generate short action chunks under receding-horizon control.
While chunk-level training is convenient across robot embodiments, it optimizes local action likelihood without explicitly accounting for long-horizon task success.
Sequence-level reinforcement learning can address this limitation, but typically requires policy rollouts and closed-loop interaction, which are costly for real-robot manipulation.
We introduce DriftOPD, a teacher-free, rollout-free framework for sequence-level on-policy distillation of continuous VLA action experts.
We show that the sequence-level reverse Kullback-Leibler (KL) divergence decomposes into a chunk-level reverse-KL term and a future-potential term that captures the long-horizon effect of the current action.

Preliminary

On-policy distillation (OPD) minimizes the reverse Kullback-Leibler divergence (KLD) between a model distribution $q_{\theta}$ and a target distribution $p$ over model-generated sequences ( Gu et al., 2024 ) : where $c$ is the conditioning context and $\bm{x}=(x_{1},\ldots,x_{T})$ is an autoregressively generated sequence.

Method

We formulate VLA learning through the reverse KLD between the distributions over action sequences induced by the model and target policies. This sequence-level perspective is illustrated in Figure 2 . Let $t$ index the action-chunk sequence, with $\bm{s}_{t}$ denoting the current VLA context (e.g., observation, instruction, and action state) and $a_{t}$ the corresponding action chunk.

Experiment

We experimentally evaluate and analyze the effectiveness of DriftOPD on robot manipulation tasks.

We use the $\pi_{0.5}$ ( Black et al., 2025a ) , GR00T N1.5/N1.6 ( Bjorck et al., 2025 ) , and ABot-M0 ( Yang et al., 2026a ) vision language action models, covering fully finetuned models, LoRA ( Hu et al., 2022 ) finetuned models, models finetuned on selected VLM layers, and a JiT ( Li and He, 2026 ) -style model that directly predicts clean actions.

Experiment — DriftOPD: Sequence-Level Reverse-KL Distillation for One-Step VLA Policies — Figure 4: GPU time to success: policy calls $\times$ per-call latency (idle B200, batch 1) (Bars: total GPU time per success, including failed episodes; Diamonds (parenthesized): mean over successful episodes).

Sources and Demonstrations

Figures are reproduced from Jun et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.