ACG-WAM: World-Action Modeling via Action-Conditioned Geometric Latent Prediction

Summary

World action models jointly learn visual predictionand robot actions, providing a way to use observations ofscene evolution for policy learning. Their video and actionlosses, however, provide no explicit target for the geometricconsequences of a demonstrated action sequence.

Abstract Summary

Key Points

• World action models jointly learn visual predictionand robot actions, providing a way to use observations ofscene evolution for policy learning.
• Their video and actionlosses, however, provide no explicit target for the geometricconsequences of a demonstrated action sequence.
• Moreover,visual features taken after temporal attention can contain futureobservations, making them unsuitable as the sole current visualinput to an auxiliary predictor.
• We introduce ACG-WAMand its auxiliary objective, the Action-Conditioned GeometricJoint-Embedding Predictive Architecture (ACG-JEPA), whichpredicts geometric features at several horizons from the currentobservation and intervening actions, using the future slot of afrozen VGGT encoding of each current and future image pairas the target.
• We apply this supervision from the head and wristcameras to a shared visual embedding before temporal mixing,and remove the teacher and auxiliary modules at inference.On 50 RoboTwin 2.0 tasks, ACG-WAM achieves 93.46%success in clean scenes, with the best randomized success(92.68%) and mean across both settings (93.07%) among thecompared methods; across three tasks on a real robot, itachieves 85.00% success and 91.67% partial completion score,exceeding Motus by 10.00 and 9.17 percentage points, respec-tively.

Method

ACG-WAM augments Motus with an auxiliary task that predicts geometric features from the current observation and a sequence of actions. At time $t$, the policy receives camera observations $\{I_{t}^{v}\}_{v\in\mathcal{V}}$, robot state $\mathbf{s}_{t}$, and a language instruction $\ell$, and generates a chunk of $H$ actions.

The Motus backbone [ 5 ] uses Mixture-of-Transformers (MoT) attention to couple visual understanding, latent video generation [ 25 ] , and action generation.

Experiments

We evaluate ACG-WAM on RoboTwin 2.0 and on three tasks with a real robot. The main comparisons measure task success, and the ablations compare prediction targets, action conditioning, and temporal horizons. The backbone uses Wan2.2-TI2V-5B and Qwen3-VL-2B-Instruct [ 29 , 30 ] . Frozen VGGT targets are cached offline following Sec. III-C , resizing wrist crops to the head-view size before preprocessing.

Sources and Demonstrations

Figures are reproduced from Liu et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.