Robotics Research Hub
Cross-Embodiment Robot Foundation World Models with Latent Actions

Cross-Embodiment Robot Foundation World Models with Latent Actions

Huang Huang, Sriram Yenamandra, Arjun Majumdar, Elie Aljalbout, Tushar Nagarajan, Tsung-Yen Yang, Akshara Rai, Michael Rabbat, Li Fei-Fei, Jiajun Wu, Tingfan Wu, Franziska Meier · N/A · 2026

manipulation reinforcement-learning benchmark
PDF arXiv

Framework

N/A

License

N/A

Stars

N/A

Summary

The diversity of robot embodiments and action spaces makes it challenging to build robot world models that generalize across different embodiments.

Abstract Summary

The diversity of robot embodiments and action spaces makes it challenging to build robot world models that generalize across different embodiments. We introduce the Latent Action-Conditioned Robot World Model (LAC-WM), which operates within a learned unified latent action space shared across diverse embodiments. This unified action space improves the world model's performance when adapted to previously unseen robot embodiments. We compare LAC-WM with an Explicit Action-Conditioned World Model (EAC-WM), which conditions on explicit motion labels. Our results show that explicit action conditioning leads to disjoint action representations across embodiments, limiting downstream performance when adapting to new robots. We evaluate both models on dexterous manipulation tasks and a modified LIBERO benchmark. LAC-WM improves downstream performance over EAC-WM by up to 46.7% on dexterous manipulation and 11.7% on LIBERO. Crucially, the unified latent action space allows LAC-WM's downstream performance to scale positively with the number of embodiments used during pretraining. In contrast, the disjoint action space in EAC-WM leads to decreased performance as the number of pretraining embodiments increases. These results highlight the importance of a unified action space for efficient cross-embodiment learning, addressing a key challenge in robotics. Project website: https://lacwm.github.io/

Key Points

  • The diversity of robot embodiments and action spaces makes it challenging to build robot world models that generalize across different embodiments.
  • We introduce the Latent Action-Conditioned Robot World Model (LAC-WM), which operates within a learned unified latent action space shared across diverse embodiments.
  • This unified action space improves the world model's performance when adapted to previously unseen robot embodiments.
  • We compare LAC-WM with an Explicit Action-Conditioned World Model (EAC-WM), which conditions on explicit motion labels.
  • Our results show that explicit action conditioning leads to disjoint action representations across embodiments, limiting downstream performance when adapting to new robots.

Method

LAC-WM is pretrained on cross-embodiment datasets and finetuned to adapt to previously unseen robot embodiments. The overall framework is in Figure 1 . During pretraining (left, yellow), LAC-WM consists of an inverse dynamics model (IDM), a forward dynamics model (FDM), and a motion decoder (MD). The IDM encodes image observations into a latent action, which conditions the FDM to predict the next image latent.

Method — Cross-Embodiment Robot Foundation World Models with Latent Actions
Figure 1 : LAC-WM Architecture . During pretraining (yellow area), the inverse dynamics model (IDM) encodes cross-embodiment image observations into a unified latent action space, as the condition for the forward dynamics model (FDM). A motion decoder translates the latent actions into motion labels, including delta human hand poses, robot end-effector actions, and camera motion. During finetuning (green area), an action projector encodes raw robot actions of unseen embodiments into the pre-trained unified latent action space.

Experiments

We use a robot world model with explicit action encoders as our baseline. Specifically, we adopt the same FDM architecture but generate action conditioning by directly encoding explicit motions, such as robot end-effector (EE) actions or delta human hand poses into action embeddings.

Since different embodiments have distinct action spaces, we employ separate action encoders for each embodiment, following common practice in multi-embodiment learning ( NVIDIA et al., 2025b ) .

Experiments — Cross-Embodiment Robot Foundation World Models with Latent Actions
Figure 2 : UMAP visualization of 7,000 action embeddings from three datasets: Droid (red), Agibot (green), and Egodex (blue). 1 : Embeddings from separate Explicit Action Encoders (EAE), which are distinctly separated by embodiment. 2 : Embeddings from a shared EAE (EAE*), still showing dataset-level separation. 3 : Embeddings from IDM without the motion decoder, where Agibot and Egodex cluster together but separate from Droid. 4 : Embeddings from the IDM, showing strong cross-dataset alignment.

Latent Action Analysis

This section addresses the first research question of Do latent action models better unify cross-embodiment action representations compared to explicit action labels? Specifically, we visualize the latent action space and qualitatively evaluate the action transfer between different embodiments.

Latent Action Analysis — Cross-Embodiment Robot Foundation World Models with Latent Actions
Figure 3 : Cross-embodiment action transfer results : The left panels illustrate robot-to-human transfer, while the right panels show human-to-robot transfer. The first row presents reference videos. Rows 2–4 display FDM generated videos using embeddings from EAE, IDM without MD, and IDM, respectively. Blue dotted lines trace end-effector trajectories from the references and are aligned across generated videos except for initial positions. IDM embeddings yield trajectories that closely match the references, whereas the other methods exhibit inaccurate motion patterns and visual artifacts (highlighted by red circles).

Robot Task Evaluation

This section studies the world model performance when adapting to a previously unseen embodiment. We quantitatively evaluate action conditioned image generation performance of the world models by predicting eight future frames based on an initial observation and an action sequence, as in Figure 6 .

For evaluation, we use the held out BFA splits to test generalization to (1) unseen instances of objects within known categories ( Unseen Instances ) and (2) three entirely unseen object categories: donut, lime, and banana ( Unseen Categories ).

Robot Task Evaluation — Cross-Embodiment Robot Foundation World Models with Latent Actions
Figure 4 : We show the image latent embedding distance $\delta f$ (left) and $\delta f_{g}$ (middle) and task success rate S.R. (right), for LAC-WM (blue) and EAC-WM (orange) pre-trained with different number of embodiments. The x axis shows the number of embodiments and dataset size used in pre-training, from left to right of the x axis, the model is trained with Egodex only, Egodex and Agibot, and Egodex, Agibot and Droid dataset.

mpact Statement

This paper presents work whose goal is to advance the field of robot world model learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here. We conduct pretraining on three datasets: EgoDex ( Hoque et al., 2025 ) , Agibot ( Bu et al., 2025a ) , and Droid ( Khazatsky et al., 2024 ) .

The action space includes changes in end-effector poses and camera poses. The end-effector pose combines the wrist pose with additional dimensions for gripper or hand articulation.

mpact Statement — Cross-Embodiment Robot Foundation World Models with Latent Actions
Figure 5 : Experiments of scaling number of embodiments or data samples. The first two points shows the scaling from two embodiments (Egodex and Droid) to three embodiments (Egodex, Droid and Agibot) with same amount of trajectories, while the last two points shows the scaling from 24k trajectories to 150k trajectories across three embodiments.

Sources and Demonstrations

Figures are reproduced from Huang et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.

Share

Related Papers

A Sim-to-Real Integration Pipeline for Training and Deployment of Chunk-Based VLA Manipulation Policies
arXiv preprint

A Sim-to-Real Integration Pipeline for Training and Deployment of Chunk-Based VLA Manipulation Policies

Mathilde Kappel, Clémence Grislain, Mohamed Chetouani et al.

Vision-Language-Action (VLA) models have become a prominent paradigm for mapping multimodal inputs, including semantic instructions, visual observations of the scene, and proprioceptive observations, to robot actions.

manipulation vision vla reinforcement-learning sim-to-real control learning-from-demonstration benchmark
PDF Advanced
A Two-Echelon Covering Tour Vehicle Routing Problem with Drones for Post-Disaster Relief
arXiv preprint

A Two-Echelon Covering Tour Vehicle Routing Problem with Drones for Post-Disaster Relief

Dang Viet Anh Nguyen, Rajesh Piplani, Aldy Gunawan

We introduce the two-echelon covering tour vehicle routing problem (2E-CTVRP) for the distribution of relief supplies after a disaster. In the first echelon, a fleet of trucks transports supplies and drones from a central depot to satellites at the periphery of the affected area.

manipulation reinforcement-learning benchmark
Code PDF Intermediate
GitHub ★ —
An Analysis of Streaming Deep Reinforcement Learning for Adaptive Continual Learning in Robotics
arXiv preprint

An Analysis of Streaming Deep Reinforcement Learning for Adaptive Continual Learning in Robotics

Teeratham Vitchutripop, Alyssa Quarles, Wenhe Zhang et al.

Over the course of a lifetime, robots may encounter novel scenarios unaccounted for in its original training that result in performance degradation. One common approach to mitigating this issue is to further grow the offline training dataset in hopes of producing a policy robust to these changes.

manipulation locomotion reinforcement-learning benchmark
PDF Intermediate
Behavioral Cloning Mystery
arXiv preprint

Behavioral Cloning Mystery

Seohong Park, Sergey Levine

Behavioral cloning (BC), despite its simplicity, exhibits many counterintuitive phenomena in the real world. For example, the performance of BC often keeps increasing as the model overfits more to the dataset, and fully closed-loop policies often completely fail without action chunking.

manipulation reinforcement-learning control learning-from-demonstration benchmark
Code PDF Intermediate
GitHub ★ —