Perceive, Predict, and Plan: a Unified World Action Model for Egocentric Dexterous Manipulation

Zirui Wu, Chen Guo, Tianjian Jiang, Zeren Jiang, Zeming Li, Lingchen Yang, Longteng Duan, Ying Xue, Handi Yin, Shenyuan Gao, Martin R. Oswald, Jun Zhang, Ping Tan, Jie Song · N/A · 2026

Summary

Abstract Summary

We introduce P$^{3}$-WAM, a unified world action model for egocentric dexterous manipulation that learns 3D perception, action-conditioned prediction, and goal-directed planning through three complementary modes: Perceive, which reconstructs articulated hand motion and camera trajectories from video; Predict, which generates visual futures under prescribed actions; and Plan, which jointly generates action trajectories and their anticipated visual consequences conditioned on an initial image, a language instruction, and an optional visual goal. Our model integrates a compact, structured 3D action representation into a pretrained video diffusion transformer through a mixture-of-transformers architecture with shared base weights, modality-specific adaptation, and bidirectional visual-action attention. Randomized modality visibility lets video and 3D actions serve as either conditioning context or prediction targets across the three modes. Experiments show that unified training improves all three capabilities over single-mode training, including gains in planning quality under a vision-language model evaluation. Through unified training, our model outperforms specialized baselines on 3D hand reconstruction and action-conditioned video prediction across four datasets, while producing goal-directed manipulation videos with stronger instruction adherence and action progression than video-generation baselines. Project Page: https://p3-wam.pages.dev/

Key Points

We introduce P$^{3}$-WAM, a unified world action model for egocentric dexterous manipulation that learns 3D perception, action-conditioned prediction, and goal-directed planning through three complementary modes: Perceive, which reconstructs articulated hand motion and camera trajectories from video; Predict, which generates visual futures under prescribed actions; and Plan, which jointly generates action trajectories and their anticipated visual consequences conditioned on an initial image, a language instruction, and an optional visual goal.
Our model integrates a compact, structured 3D action representation into a pretrained video diffusion transformer through a mixture-of-transformers architecture with shared base weights, modality-specific adaptation, and bidirectional visual-action attention.
Randomized modality visibility lets video and 3D actions serve as either conditioning context or prediction targets across the three modes.
Experiments show that unified training improves all three capabilities over single-mode training, including gains in planning quality under a vision-language model evaluation.
Through unified training, our model outperforms specialized baselines on 3D hand reconstruction and action-conditioned video prediction across four datasets, while producing goal-directed manipulation videos with stronger instruction adherence and action progression than video-generation baselines.

Method

We consider an egocentric video $\mathbf{V}=\{\mathbf{I}_{t}\}_{t=1}^{T}$ with aligned hand states $\mathbf{H}=\{\mathbf{h}_{t}\}_{t=1}^{T}$ and camera states $\mathbf{P}=\{\mathbf{P}_{t}\}_{t=1}^{T}$, which encode rotation and translation relative to the first frame. We define the 3D action sequence as $\mathbf{A}=\{\mathbf{a}_{t}\}_{t=1}^{T}=\{\langle\mathbf{h}_{t},\mathbf{P}_{t}\rangle\}_{t=1}^{T}$.

Action tokens $\mathbf{z}_{a}$ encode the aligned hand and camera observations.

Method — Perceive, Predict, and Plan: a Unified World Action Model for Egocentric Dexterous Manipulation — Figure 1: P 3 -WAM unifies perception, prediction, and planning. Perceive recovers hand motion from video; Predict generates video under prescribed motion; Plan jointly generates motion and video guided by text instructions and optional visual goals.

Experiments

We first examine whether unified training benefits each mode (Sec. 4.1 ), then compare Perceive , Predict , and Plan with specialized baselines on their respective tasks. We also evaluate whether the planned human actions can be transferred to a real-world dexterous robot (Sec. 4.4 ).

We initialize the single-mode and unified models from Wan2.2-Fun A14B-Control and evaluate them at $5{,}000$ total training steps, with all other training variables held fixed.

Experiments — Perceive, Predict, and Plan: a Unified World Action Model for Egocentric Dexterous Manipulation — Figure 2: Task modes and unified training. Colored tiles denote observed video $\mathbf{V}$ and actions $\mathbf{A}$; gray tiles denote missing inputs. Unified training samples these patterns across clips and time.

AI Use Disclosure

Generative AI assisted with manuscript drafting and editing, figure development, and research-code development and debugging. AI-generated images also serve as planning inputs in the examples identified as generated in the paper. A VLM provides evaluation scores under the protocol in Appendix E ; human ratings are collected separately.

The authors are responsible for reviewing the assisted material and for the accuracy, originality, and attribution of the final submission.

AI Use Disclosure — Perceive, Predict, and Plan: a Unified World Action Model for Egocentric Dexterous Manipulation — Figure 3: P 3 -WAM couples video and action streams through joint attention and shared base weights. The action expert adds low-rank adaptation and directly predicts hand and camera states.

Reproducibility Statement

Section 3 and Appendices A – C describe the architecture, objectives, codec, and training. Appendix D specifies preprocessing, baseline interfaces, and metrics; Appendix E gives the human/VLM rubric and bootstrap procedure. Additional measurements appear in Appendix F . Within each diffusion expert, RGB and action use the same base self-attention Q/K/V and output projections and the same two FFN projections.

The action branch adds low-rank updates to these six projections and to the text cross-attention Q and output projections.

Reproducibility Statement — Perceive, Predict, and Plan: a Unified World Action Model for Egocentric Dexterous Manipulation — Figure 4: Visual goals specify spatial targets. (a) A goal image specifies how far apart the red blocks should be. (b) Intermediate goal images guide the block arrangement at each stage of a longer task; without them, placement errors accumulate.

Sources and Demonstrations

Figures are reproduced from Wu et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.