Native Video-Action Pretraining for Generalizable Robot Control

Qihang Zhang, Lin Li, Luyao Zhang, Shuai Yang, Yiming Luo, Shuaiting Li, Ruilin Wang, Junke Wang, Jiahao Shao, Gangwei Xu, Jiaming Zhou, Yishu Shen, Yudong Jin, Fangyi Xu, Shuailei Ma, Jiaqi Liao, Guanxing Lu, Zifan Shi, Yongkun Wen, Yujie Zhao, Weixuan Tang, Xinyang Wang, Chaojian Li, Jiapeng Zhu, Ka Leong Cheng, Nan Xue, Xing Zhu, Yujun Shen, Yinghao Xu · N/A · 2026

Summary

Abstract Summary

The advent of video-action models offers a promising path for robot control. Nevertheless, we argue that repurposing video generative models designed for digital content creation is inherently inadequate for physical environments. To bridge this gap, we present LingBot-VA 2.0, a video-action foundation model built from the ground up for embodiment. Four core design principles showcase its evolution from LingBot-VA. (1) Departing from traditional reconstruction-focused VAEs, we introduce a semantic visual-action tokenizer, which aligns visual representations with both semantics and actions, improving instruction following and action precision in subsequent policy learning. (2) Given the strictly causal nature of temporal dynamics, we adopt a causal pretraining paradigm, training from scratch to circumvent the catastrophic forgetting that frequently occurs when adapting bidirectional architectures. (3) To meet the demands of high-frequency inference, our model employs a sparse MoE backbone, expanding model capacity without compromising efficiency. (4) Real-time closed-loop control is realized through an enhanced asynchronous inference scheme, which predicts future latents in parallel with action execution while re-grounding each rollout on the latest observation via learned forward dynamics. Real-world deployment validates LingBot-VA 2.0 as a robust foundation model, as evidenced by its few-shot generalization across complex manipulation tasks.

Key Points

The advent of video-action models offers a promising path for robot control
Nevertheless, we argue that repurposing video generative models designed for digital content creation is inherently inadequate for physical environments.
To bridge this gap, we present LingBot-VA 2.0, a video-action foundation model built from the ground up for embodiment.
Four core design principles showcase its evolution from LingBot-VA
(1) Departing from traditional reconstruction-focused VAEs, we introduce a semantic visual-action tokenizer, which aligns visual representations with both semantics and actions, improving instruction following and action precision in subsequent policy learning.

Native Video-Action Model

This section presents the full LingBot-VA 2.0 stack ( ). After reviewing how prior video-action models repurpose generic video generators for control ( ), we build the stack natively in two stages. The first stage trains a semantic visual-action tokenizer, which aligns visual latents with a frozen foundation model and extracts latent actions from unlabeled video, yielding a shared visual-action latent space ( ).

Data Recipe

LingBot-VA 2.0 is trained with a data recipe that follows the multi-task curriculum of : general text-to-image pretraining, general text-to-video pretraining, and finally video-action adaptation on embodied data. For the general text-to-image and text-to-video pretraining stages, we reuse the image and video corpora curated for LingBot-Video [ 70 ] .

These web-scale image and video data provide broad appearance and dynamics priors before video-action adaptation.

Data Recipe — Native Video-Action Pretraining for Generalizable Robot Control — Figure 2 : Semantic visual-action tokenizer . The visual tokenizer aligns reconstruction-oriented latents with visual foundation-model features, while the latent action tokenizer learns compact transition variables through inverse and forward dynamics. Adapted from RepWAM [ 98 ] .

Experiments

Our model is a diffusion transformer (DiT) trained from scratch, operating on the latent space of our semantic visual-action tokenizer (96 latent channels). Video latents are patchified with a $1\times 2\times 2$ ($T\times H\times W$) linear patch embedding into a stream of visual tokens.

The video backbone consists of 30 transformer blocks with video hidden dimension 2048; joint self-attention is performed in a 3072-dim attention space (24 attention heads $\times$ 128 head dim).

Experiments — Native Video-Action Pretraining for Generalizable Robot Control — Figure 3 : Dual-system hierarchical policy . A high-level VLM planner consumes the task goal and sparse recent observations to produce structured subtask context, which conditions the low-level video-action policy together with robot state to produce closed-loop actions.

Contributors

Qihang Zhang, Lin Li, Junke Wang, Jiahao Shao, Gangwei Xu, Luyao Zhang, Jiaming Zhou, Yudong Jin, Shuailei Ma, Jiaqi Liao, Ka Leong Cheng Pretraining Data: Lin Li, Luyao Zhang, Qihang Zhang, Jiaming Zhou, Yudong Jin, Xinyang Wang Posttraining: Luyao Zhang ∗ , Shuai Yang ∗ , Lin Li, Qihang Zhang, Gangwei Xu, Jiapeng Zhu, Yujie Zhao, Weixuan Tang Acceleration: Shuaiting Li, Chaojian Li Deployment & Demo: Yiming Luo ∗ , Shuai Yang ∗ , Ruilin Wang, Yishu Shen, Jiahao Shao, Fangyi Xu, Shuaiting Li, Guanxing Lu, Zifan Shi, Yongkun Wen Project Supervi...

Sources and Demonstrations

Figures are reproduced from Zhang et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.