DreamX-Phi 1.0: Action-Conditioned Video World Model for Robotic Manipulation

DreamX Team, Rui Chen, Xiangxiang Chu, Geng Li, Jifan Li, Qingfeng Shi, Datao Tang, Jing Tang, Jun Wang, Pengfei Zhang · N/A · 2026

Summary

Abstract Summary

We present \textbf{DreamX-Phi 1.0}, an action-conditioned video world model for robotic manipulation that, given an observed frame, a language instruction, and a prescribed action sequence comprising end-effector poses and gripper states, predicts the resulting future observations. Yet realism alone does not guarantee faithfulness: a convincing rollout can still move the wrong arm or lose the manipulated object. To ensure the prediction respects each arm's commanded path, we inject per-arm $\mathrm{SE}(3)$ transformations into attention via \textbf{PRoPE-style geometric encoding}, preserving arm identity and rigid-motion structure. Action control alone does not fully constrain scene geometry or the evolution of small manipulated objects. We therefore add a lightweight \textbf{depth branch} for scene-level geometry and use \textbf{SAM3 masks} with a frozen \textbf{V-JEPA teacher} to maintain object consistency throughout grasping. We further distill the multi-step generator into a few-step student via distribution-matching distillation for efficient deployment. At the time of writing, \model{} achieves first place on Track~1 and second place on Track~2 of the WorldArena~2.0 Challenge. Our model and code will be publicly available.

Key Points

We present \textbf{DreamX-Phi 1.0}, an action-conditioned video world model for robotic manipulation that, given an observed frame, a language instruction, and a prescribed action sequence comprising end-effector poses and gripper states, predicts the resulting future observations.
Yet realism alone does not guarantee faithfulness: a convincing rollout can still move the wrong arm or lose the manipulated object.
To ensure the prediction respects each arm's commanded path, we inject per-arm $\mathrm{SE}(3)$ transformations into attention via \textbf{PRoPE-style geometric encoding}, preserving arm identity and rigid-motion structure.
Action control alone does not fully constrain scene geometry or the evolution of small manipulated objects.
We therefore add a lightweight \textbf{depth branch} for scene-level geometry and use \textbf{SAM3 masks} with a frozen \textbf{V-JEPA teacher} to maintain object consistency throughout grasping.

Data Curation

Reliable action-conditioned prediction depends not only on the diversity of robot motions, but also on broad visual coverage and consistent alignment between observations and control signals. We therefore construct the corpus from three complementary sources: action-free egocentric video, real-robot demonstrations, and simulated robot trajectories.

Together, these sources span everyday visual dynamics, physically executed manipulation, and controlled variation in task and scene configuration. Table 1 summarizes their scale.

Data Curation — DreamX-Phi 1.0: Action-Conditioned Video World Model for Robotic Manipulation — Figure 2: Overview of the DreamX-Phi 1.0 framework. (1) Given an observed RGB frame, a language instruction, and an externally specified bimanual action trajectory, a Wan2.2-TI2V-5B video diffusion transformer predicts the future RGB video. Arm-grouped PRoPE and a robot-only optical-flow cue provide complementary geometric and image-plane action conditioning. (2) During training, SAM3-derived masks reweight the RGB objective, Depth Anything 3 (DA3) depth maps provide targets for the depth-latent objective, and a frozen V-JEPA teacher supplies object-relational supervision. (3) DMD with adversarial training distills the multi-step teacher into a few-step student.

Method

Given an observed RGB frame $\mathbf{x}_{0}$, a language instruction $\mathbf{c}$, and a prescribed bimanual action trajectory $\mathbf{a}_{1:T}$ containing end-effector poses and gripper states, our goal is to model the conditional distribution using a Wan2.2-TI2V-5B video diffusion transformer.

The latent of the first frame provides the visual context, and the future-frame latents are learned under a flow-matching objective ( Lipman et al., 2023 ) .

Method — DreamX-Phi 1.0: Action-Conditioned Video World Model for Robotic Manipulation — Figure 3: Qualitative WorldArena 2.0 Track 1 rollouts from DreamX-Phi 1.0. Each row is a predicted episode, with frames sampled in temporal order from left to right. (a) Standard RoboTwin 2.0 scenes. (b) Domain-randomized scenes with varied backgrounds, textures, lighting, and distractor objects.

Evaluation

We evaluate DreamX-Phi 1.0 along two complementary axes: the fidelity of its predicted visual rollouts and its utility as a learned environment for policy training.

WorldArena 2.0 is our primary benchmark: Track 1 evaluates video prediction conditioned on language instructions or robot actions, while Track 2 measures whether a policy optimized through interaction with the submitted world model succeeds in held-out simulator episodes.

Future Work

The current DreamX-Phi 1.0 is formulated as a Forward Dynamics Model (FDM), which predicts future observations from an externally provided action sequence and does not generate actions itself. We will extend this framework to other model formulations. In particular, we plan to develop a joint World Action Model that generates future video and robot action trajectories together.

Training these outputs jointly should help align each proposed action with its predicted visual consequences.

Sources and Demonstrations

Figures are reproduced from Team et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.