UniWAM: Unified World-Action Model

UniWAM: Unified World-Action Model

Jiayi Chen, Wenxuan Song, Jingbo Wang, Shuai Zhou, Xicheng Gong, Zehua Fan, Ziyang Zhou, Junwu E, Haodong Yan, Fuhao Li, Qize Yu, Xu Huang, Pengwei Wang, Wen Chen, Shunbo Zhou, Haoang Li · N/A · 2026

vision vla reinforcement-learning learning-from-demonstration benchmark

Framework

N/A

License

N/A

Stars

N/A

Summary

Vision-language-action models benefit from the understanding and reasoning capabilities of pretrained vision-language models, but action-only supervision provides limited grounding in world dynamics.

Abstract Summary

Vision-language-action models benefit from the understanding and reasoning capabilities of pretrained vision-language models, but action-only supervision provides limited grounding in world dynamics. Conversely, world-action models inherit spatiotemporal priors from video generation models, yet remain limited in semantic understanding and reasoning under distribution shifts. We introduce UniWAM, a unified architecture that integrates a physical reasoner, a world generator, and an action predictor to jointly learn semantic understanding of the physical world, visual generation, and action prediction. To ensure the quality of the training data, we developed a rigorous data cleaning and annotation pipeline for both human egocentric data and robot data. To adapt the vision-language component to embodied tasks while preserving its inherited language capabilities, we represent low-level actions in natural language and introduce a pre-training recipe that assigns complementary supervision from visual question answering (VQA) data, human egocentric data, and robot demonstrations to the appropriate model components. During post-training, future visual noise augmentation reduces reliance on precise future predictions, while history-conditioned flow matching uses encoded action history to initialize action generation. Together, these designs significantly reduce denoising steps while maintaining performance. UniWAM achieves state-of-the-art (SOTA) performance across multiple evaluations, including in-distribution performance, robustness, generalization, instruction following, and long-horizon task execution. Furthermore, we uncover a log-linear scaling law of unified human-robot co-training, demonstrating the effectiveness of large-scale pre-training on a mixture of human and robot data.

Key Points

  • Vision-language-action models benefit from the understanding and reasoning capabilities of pretrained vision-language models, but action-only supervision provides limited grounding in world dynamics.
  • Conversely, world-action models inherit spatiotemporal priors from video generation models, yet remain limited in semantic understanding and reasoning under distribution shifts.
  • We introduce UniWAM, a unified architecture that integrates a physical reasoner, a world generator, and an action predictor to jointly learn semantic understanding of the physical world, visual generation, and action prediction.
  • To ensure the quality of the training data, we developed a rigorous data cleaning and annotation pipeline for both human egocentric data and robot data.
  • To adapt the vision-language component to embodied tasks while preserving its inherited language capabilities, we represent low-level actions in natural language and introduce a pre-training recipe that assigns complementary supervision from visual question answering (VQA) data, human egocentric data, and robot demonstrations to the appropriate model components.

Data

Our robot pretraining data combine AgiBotWorld2026 ( AgiBot World Team, 2026 ) , Bridge ( Walke et al., 2023 ) , Droid ( Khazatsky et al., 2024 ) , Fractal ( Brohan et al., 2023 ) , Robocoin ( Wu et al., 2025 ) , and Interdata-a1 ( Tian et al., 2025 ) , covering single-arm and dual-arm embodiments. The selected data total approximately 4,363 hours, with a per-source breakdown in Table 1 .

We also apply language-action supervision to these robot data, using descriptions of local manipulation behavior.

Data — UniWAM: Unified World-Action Model
Figure 2 : EgoANT annotation pipeline. An untrimmed egocentric video is represented as timestamped contact sheets and segmented through a full-episode coarse pass followed by local temporal refinement. With the refined boundaries fixed, the multi-candidate annotation route generates descriptions using raw-frame annotation, an alternative FFmpeg decoding and frame-sampling path, and relabeling conditioned on seed or prior descriptions. A text-only selector chooses the final description without re-examining the video. The output consists of temporal intervals paired with concise subtask descriptions, which we use to annotate EgoDex.

Method

As shown in Figure 3 , UniWAM unifies semantic understanding, language modeling, visual generation, and action prediction in a Mixture-of-Transformers (MoT) architecture ( Liang et al., 2025 ) , i.e., physical reasoner, world generator, and action predictor. They exchange information through joint attention.

Method — UniWAM: Unified World-Action Model
Figure 3 : Overview of our UniWAM. (a) UniWAM comprises a physical reasoner, a world generator, and an action predictor interconnected through joint multimodal attention. Robot data and human egocentric data are jointly used to train all three experts, with supervision for the physical reasoner provided in the form of physical language. General VQA and Embodied Question Answering (EQA) data are additionally used to train the physical reasoner. (b) Overview of the pretraining data: Human and robot data are measured in hours; vision–language data are measured in millions of samples. Percentages are computed within each data category.

Experiments

We design our experiments to answer the following questions: (Q1) Can our UniWAM successfully perform manipulation tasks under in-domain conditions, complex perturbations, and OOD conditions, respectively? ( Section 5.1 ) (Q2) Does the unified design of UniWAM improve instruction following and long-horizon manipulation in the real world? ( ) (Q3) Does our unified model scale effectively with data?

( Section 5.3 ) (Q4) How does our physical language shape the learned representations and enable strong generalization? ( Section 5.3 ) (Q5) How impo...

Experiments — UniWAM: Unified World-Action Model
Figure 4 : History-conditioned action denoising. Action generation starts from perturbed action history rather than Gaussian noise.

Sources and Demonstrations

Figures are reproduced from Chen et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.

Share

Related Papers

A Sim-to-Real Integration Pipeline for Training and Deployment of Chunk-Based VLA Manipulation Policies
arXiv preprint

A Sim-to-Real Integration Pipeline for Training and Deployment of Chunk-Based VLA Manipulation Policies

Mathilde Kappel, Clémence Grislain, Mohamed Chetouani et al.

Vision-Language-Action (VLA) models have become a prominent paradigm for mapping multimodal inputs, including semantic instructions, visual observations of the scene, and proprioceptive observations, to robot actions.

manipulation vision vla reinforcement-learning sim-to-real control learning-from-demonstration benchmark
PDF Advanced
Capability-Aware Arbitration for Semantic Intent-Based Shared Control
arXiv preprint

Capability-Aware Arbitration for Semantic Intent-Based Shared Control

Zhaoda Du, Michael Bowman, Xiaoli Zhang

Shared control often allocates robot authority based on confidence in inferred human intent, assuming reliable autonomous execution. When this assumption fails, high intent confidence can cause over-helping.

manipulation vision vla reinforcement-learning control learning-from-demonstration benchmark
Code PDF Intermediate
GitHub ★ —
Co-Evolving Robot Orchestrators and Policies through Deployment
arXiv preprint

Co-Evolving Robot Orchestrators and Policies through Deployment

Xilun Zhang, Maggie Wang, Erik Bauer et al.

Vision-language-action (VLA) policies trained on large datasets are capable within their training domains, yet they still fail to generalize to the variety of situations a robot meets in real-world deployment.

vision vla reinforcement-learning learning-from-demonstration benchmark
PDF Intermediate
DexVerse: A Modular Benchmark for Multi-Task, Multi-Embodiment Dexterous Manipulation
arXiv preprint

DexVerse: A Modular Benchmark for Multi-Task, Multi-Embodiment Dexterous Manipulation

Yunchao Yao, Zhuxiu Xu, Tianqi Zhang et al.

Building general-purpose dexterous manipulation policies requires benchmarks that go beyond isolated tasks to systematically evaluate policies across diverse interaction modes, sensory conditions, and robot embodiments.

manipulation vision vla reinforcement-learning control learning-from-demonstration benchmark
PDF Advanced