GeoLAM: Learning Geometry-Grounded Latent Actions from Unlabeled Human Videos

Yifan Xie, Hekun Tian, Jinkun Liu, YuAn Wang, Qiao Sun, Wenbo Ding · N/A · 2026

Summary

Abstract Summary

Human videos provide rich manipulation experience, but extracting action representations that preserve useful motion remains challenging. Visual reconstruction alone can entangle manipulation-related motion with appearance changes and camera movement. We present GeoLAM, a framework for learning geometry-grounded latent actions from action-free human videos. GeoLAM combines future-frame reconstruction through a frozen geometric feature hierarchy with motion supervision from a training-only 4D geometry teacher. The geometric representation provides a structural prior, while the teacher's predictions yield spatially pooled targets capturing 3D displacement, residual image-plane motion, and surface-orientation changes. Visibility and confidence weighting reduces the contribution of unreliable estimates, encouraging continuous latent actions to retain geometric motion without explicit hand-pose or hand-trajectory annotations. After video pretraining without action labels, the learned representation provides transition targets for a world-action model trained on action-labeled robot demonstrations. The model jointly denoises latent actions and executable action chunks, with future-video prediction used only as an auxiliary training task. Deployment therefore requires neither the geometry teacher nor future-video generation. Evaluations on a latent-action benchmark and robotic manipulation tasks demonstrate the strong performance of GeoLAM.

Key Points

Human videos provide rich manipulation experience, but extracting action representations that preserve useful motion remains challenging.
Visual reconstruction alone can entangle manipulation-related motion with appearance changes and camera movement.
We present GeoLAM, a framework for learning geometry-grounded latent actions from action-free human videos.
GeoLAM combines future-frame reconstruction through a frozen geometric feature hierarchy with motion supervision from a training-only 4D geometry teacher.
The geometric representation provides a structural prior, while the teacher's predictions yield spatially pooled targets capturing 3D displacement, residual image-plane motion, and surface-orientation changes.

Method

Latent-action models recover transition variables from action-free videos by coupling an inverse dynamics model (IDM) with a forward dynamics model (FDM) [ 4 ] . However, image prediction alone may entangle controllable motion with appearance changes and camera motion [ 5 , 6 ] . As illustrated in Fig.

1 , we reduce this ambiguity with two complementary constraints: geometry-aware future-frame reconstruction and training-only supervision from a 4D geometry teacher. We denote the two clip endpoints by $\mathbf{I}^{t}$ and $\mathbf{I}^{t+k}$.

Method — GeoLAM: Learning Geometry-Grounded Latent Actions from Unlabeled Human Videos — Fig. 2 : Geometry-grounded latent actions for the world action model: during training, a frozen encoder supplies continuous transition targets and future-video prediction regularizes the Video DiT, whereas inference jointly denoises latent actions and action chunks using only cached current-observation key–value features. Snowflakes and flames denote frozen and trainable modules, respectively.

Experiments

Training. All models are trained with distributed data parallelism on 64 NVIDIA A800 GPUs using bfloat16 mixed precision. We optimize the latent-action model for 200,000 steps with AdamW, a learning rate of $10^{-4}$, and a per-GPU batch size of 32 (2,048 globally). RGB inputs are resized to $224\!\times\!224$, while the frozen D4RT teacher processes four uniformly sampled context frames at $256\!\times\!256$.

Sources and Demonstrations

Figures are reproduced from Xie et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.