Robotics Research Hub
CLAP: Cross-Embodiment Video World Models are Zero-Shot Physical Simulators

CLAP: Cross-Embodiment Video World Models are Zero-Shot Physical Simulators

Kechen Liu, Ola Shorinwa · N/A · 2026

reinforcement-learning
PDF arXiv Code (GitHub)

Framework

N/A

License

N/A

Stars

N/A

Summary

State-of-the-art action-conditioned video models are typically restricted to a single robot embodiment, preventing them from leveraging the vast corpus of heterogeneous video data that contains rich signals for learning generalizable physics.

Abstract Summary

State-of-the-art action-conditioned video models are typically restricted to a single robot embodiment, preventing them from leveraging the vast corpus of heterogeneous video data that contains rich signals for learning generalizable physics. To bridge this gap, we introduce CLAP, a framework for cross-embodiment action-conditioned video generation capable of being trained on diverse, internet-scale videos across human and robotic agents. CLAP is grounded in the insight that universal physical laws govern spatiotemporal dynamics regardless of the actor. However, cross-embodiment learning is non-trivial because action representations vary sharply across robot platforms and are typically absent in human videos. CLAP addresses this fundamental challenge through the following core contributions. First, CLAP reconciles disparate action spaces using end-effector poses, language instructions, and latent actions. Second, to resolve their individual limitations, CLAP introduces a curriculum-based cross-embodiment learning recipe that first learns foundational physical priors across unlabeled video data using latent actions and subsequently grounds them in end-effector action spaces for zero-shot deployment to real-world tasks. Crucially, CLAP approaches or surpasses state-of-the-art single-embodiment video models in challenging environments like DROID. These performance advantages compound via few-shot adaptation to establish a novel paradigm for training single-embodiment video world models. Ultimately, CLAP delivers the most comprehensive suite of action-conditioned video world models to date - spanning diverse action-conditioning spaces (end-effector, language, and latent) and robot morphologies (including cross-embodiment, DROID, Bridge, bimanual YAM robots, and G1 humanoids). We open-source all code and models. Project Website at https://omni-clap.github.io .

Key Points

  • State-of-the-art action-conditioned video models are typically restricted to a single robot embodiment, preventing them from leveraging the vast corpus of heterogeneous video data that contains rich signals for learning generalizable physics.
  • To bridge this gap, we introduce CLAP, a framework for cross-embodiment action-conditioned video generation capable of being trained on diverse, internet-scale videos across human and robotic agents.
  • CLAP is grounded in the insight that universal physical laws govern spatiotemporal dynamics regardless of the actor.
  • However, cross-embodiment learning is non-trivial because action representations vary sharply across robot platforms and are typically absent in human videos.
  • CLAP addresses this fundamental challenge through the following core contributions

CLAP: Training Cross-Embodiment Video World Models

Cross-embodiment learning from video data presents a unique challenge: robot and human data is inherently heterogeneous with different observation and action spaces, and internet-scale video data largely lacks action labels.

To address these challenges, CLAP introduces a novel recipe for training cross-embodiment video world models, reconciling heterogeneous action spaces using end-effector poses, natural language, and latent action representations (see Secs. 3.1 and 3.2 ).

CLAP: Training Cross-Embodiment Video World Models — CLAP: Cross-Embodiment Video World Models are Zero-Shot Physical Simulators
Figure 2: CLAP CLAP reconciles disparate action spaces across human and robot morphologies to train high-fidelity cross-embodiment video models.

Experiments

Through the lens of the following research questions, we investigate the impact of cross-embodiment scaling on action-conditioned video models and their efficacy in downstream robotics tasks: [Q1.] Do cross-embodiment video models surpass their single-embodiment counterparts? ( Sec. 4.1 ) [Q2.] Do relative-action spaces outperform absolute-action spaces in cross-embodiment video modeling? ( Sec.

4.2 ) [Q3.] Do cross-embodiment video models achieve zero-shot real-world generalization? ( Sec.

Experiments — CLAP: Cross-Embodiment Video World Models are Zero-Shot Physical Simulators
Figure 3: Comparison to SOTA single-embodiment baselines. CLAP approaches or exceeds SOTA performance on the DROID platform, showing only minor fidelity degradation in simpler domains like Bridge. Across the diverse mix of embodiments (OXE-Mix), CLAP-LAM and CLAP-CURR achieve the highest prediction accuracies, closely followed by CLAP-EE .

Preliminaries

Video world models predict future outcomes conditioned on per-frame actions, starting from an initial camera observation. Furthermore, to capture temporal context such as robot velocities that dictate future evolution, video world models are often conditioned on a history of past observations.

Preliminaries — CLAP: Cross-Embodiment Video World Models are Zero-Shot Physical Simulators
Figure 4: CLAP’s performance on DROID. CLAP-CURR combines the strengths of CLAP-LAM and CLAP-EE to achieve high-fidelity dynamics prediction with fine-grained per-frame controllability, outperforming SOTA single-embodiment baselines. Meanwhile, CLAP-LANG delivers comparable performance over shorter horizons, though it is constrained by compounding errors over longer trajectories.

Method

End-effector Actions. Each demonstration is stored as a trajectory of end-effector states aligned to the video frames.

A single state is a $7$-dimensional vector ${s_{t}=\big[\,x_{t},\;y_{t},\;z_{t},\;\phi_{t},\;\theta_{t},\;\psi_{t},\;g_{t}\,\big],}$ where $(x_{t},y_{t},z_{t})$ is the gripper position in meters in the robot base frame, $(\phi_{t},\theta_{t},\psi_{t})$ are the roll–pitch–yaw orientation Euler angles in radians, and $g_{t}$ is the raw gripper signal.

Method — CLAP: Cross-Embodiment Video World Models are Zero-Shot Physical Simulators
Figure 5: CLAP’s performance on Bridge. Relative to the SOTA baseline, CLAP achieves high-accuracy future prediction on the Bridge platform; however, the performance of CLAP-LANG degrades over extended prediction horizons.

Experiments

CLAP trains cross-embodiment video models on diverse datasets using the sampling ratios in Tab. 3 , proportional to the complexity and quaity of the individual datasets.c Multi-view camera images are vertically stacked into a single fixed-size frame (${576\times 320}$) after resizing individual frames to ${192\times 320}$.

For uniformity, we repeat single-view camera inputs in datasets without multi-view images, although preliminary experiments revealed that such repetition is unnecessary.

Experiments — CLAP: Cross-Embodiment Video World Models are Zero-Shot Physical Simulators
Figure 6: Comparison between relative-action and absolute-action spaces. Action-conditioned video generation in absolute-action spaces yields superior prediction fidelity; however, these gains are limited to future prediction tasks conditioned on end-effector-actions. In contrast, relative actions achieve higher prediction fidelity for language-action-conditioned video models.

Nuanced Summary

This section discusses the nuances surrounding CLAP’s novelty, broader potential impact, and scope of claims. CLAP uses an old video backbone (SVD, 2023), a pretrained language encoder (CLIP), simple MLP action encoders, and does not introduce a new dataset. Is it novel? Answer: Not quite. By that same token, current state-of-the-art action-conditioned video models are largely derivative.

CLAP’s novelty lies in introducing a new paradigm for action-conditioned video modeling rooted specifically in cross-embodiment learning.

Nuanced Summary — CLAP: Cross-Embodiment Video World Models are Zero-Shot Physical Simulators
Figure 7: Comparison between relative-action and absolute-action spaces in DROID. In end-effector-action-conditioned video generation, absolute-action spaces enable fine-grained controllability over extended prediction horizons, an advantage that is not observed with relative-action spaces. Conversely, language-action-conditioned video models achieve higher-accuracy future prediction with relative actions.

Sources and Demonstrations

Figures are reproduced from Liu et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.

Share

Related Papers

4D Radar Perception Algorithms for Autonomous Driving: A Review
arXiv preprint

4D Radar Perception Algorithms for Autonomous Driving: A Review

Xumin Wu, Jun Zhou, Jilin Mei et al.

Research on 4D millimeter-wave radar perception algorithms has flourished in recent years, extending from signal processing and object detection to semantic segmentation, motion estimation, occupancy prediction, and dynamic scene reconstruction.

vision reinforcement-learning benchmark
PDF Beginner