Modality-Autoregressive World-Action Models

Adam Hung, Bardienus P. Duisterhof, Deva Ramanan, Jeffrey Ichnowski · N/A · 2026

Summary

Abstract Summary

World-action models (WAMs) jointly model future observations and actions, typically predicting the future as RGB images. Other visual modalities such as depth, pretrained visual features, and point tracks can more efficiently capture geometric, semantic, and motion features. However, how best to combine these modalities within WAMs remains an open question. We introduce ModAR, the first WAM to autoregressively denoise multiple future modalities before predicting actions. This allows each prediction to condition on previously generated modalities. We train from scratch to systematically study how training-data mixtures, predicted modalities, and WAM formulations affect performance. In our evaluations, WAMs benefit from predicting point tracks, DINO features, and depth maps, while additionally predicting future RGB does not provide a consistent benefit. We also find that ModAR's sequential generation outperforms existing WAM formulations, with the highest average success rate at all evaluated data scales. We also fine-tune the video-model-initialized WAM Flex-$π$ on the same data; ModAR achieves a slightly higher observed average success rate (75% vs. 72%) while using approximately $20\times$ fewer training FLOPs and no pretraining. On three real-world bimanual tasks, ModAR outperforms baselines and improves with human videos.

Key Points

World-action models (WAMs) jointly model future observations and actions, typically predicting the future as RGB images.
Other visual modalities such as depth, pretrained visual features, and point tracks can more efficiently capture geometric, semantic, and motion features.
However, how best to combine these modalities within WAMs remains an open question
We introduce ModAR, the first WAM to autoregressively denoise multiple future modalities before predicting actions.
This allows each prediction to condition on previously generated modalities

Method

Given demonstrations spanning multiple manipulation tasks, our objective is to jointly model future multimodal observations and robot actions conditioned on the current observation, robot configuration, and task label. We consider the set of predicted future modalities $\mathcal{M}=\{\mathrm{rgb},\mathrm{depth},\mathrm{dino},\mathrm{tracks}\}$.

At decision time $t$, let $\mathbf{o}_{t}$ denote the multimodal observation.

Method — Modality-Autoregressive World-Action Models — Fig. 5: Evaluating predicted futures with a separate IDM. Recall from Sec. III that ModAR’s final action-prediction step acts as an inverse-dynamics model (IDM), mapping its future-observation predictions to actions. In contrast, Unified denoises futures and actions jointly, so its action prediction conditions on partially noisy predicted futures. To remove this asymmetry, we discard both models’ native action predictions and pass their predicted futures to the same separately trained IDM. ModAR still outperforms Unified, showing that its predicted futures are themselves more useful for predicting actions.

Experiments

In simulation, we evaluate WAM formulations (Figure 3 ), actionless-data scaling, and predicted modality sets. We also compare with a large video-model-initialized WAM, Flex-$\pi$ [ 9 ] . We then evaluate ModAR on real-world bimanual manipulation and learning from human demonstrations. Simulation.

We evaluate on six representative RoboTwin [ 32 ] tasks: “dump bin”, “pick bottles”, “place bread”, “put bottles”, “stack bowls”, and “turn switch.” Each task uses $D\in\{50,250,1250\}$ total training demonstrations.

Experiments — Modality-Autoregressive World-Action Models — Fig. 6: ModAR ablations in RoboTwin. Average success rate over six tasks, with 250 total demonstrations per task and 50 action-labeled demonstrations. We ablate context noise ( No context noise ), generation order ( Reverse order ), and each individual future-observation modality. Results suggest that context noise is critical for robust autoregressive generation. Generating modalities in reverse order also reduces performance, supporting our hypothesis that more structured modalities act as “scratchpads” for more detailed modalities. Removing future RGB prediction does not reduce performance, suggesting that RGB contributes the least of the four future-observation modalities.

Sources and Demonstrations

Figures are reproduced from Hung et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.