SkeleWAM: Skeleton World-Action Modeling for Efficient Robotic Manipulation

Summary

Abstract Summary

World action models (WAMs) combine robot action generation with future state prediction. Existing WAMs typically predict videos or learned visual latents, which represent interaction geometry only implicitly and may retain appearance information unrelated to control. We introduce SkeleWAM, a compact WAM that represents a manipulation scene as a sparse 3D skeleton composed of robot joints, object centers, and interaction points. Constructed online from current RGB-D observations and robot proprioception, the skeleton provides a unified geometric state for action generation and future skeleton prediction. Future skeleton prediction provides additional geometric supervision for action learning without requiring visual reconstruction. At inference, SkeleWAM generates actions directly from the current skeleton and language instruction, while Medoid Action Consensus (MAC) serves as an auxiliary consensus strategy for stochastic action samples. On LIBERO-Plus, SkeleWAM achieves an overall success rate of 85.9% with 57.1M parameters, outperforming Cosmos-Policy by 3.7 percentage points. These results demonstrate that sparse 3D robot--object structure provides an effective state space for robust and parameter-efficient world action learning. The project is available at https://skelewam-project.github.io/.

Key Points

World action models (WAMs) combine robot action generation with future state prediction
Existing WAMs typically predict videos or learned visual latents, which represent interaction geometry only implicitly and may retain appearance information unrelated to control.
We introduce SkeleWAM, a compact WAM that represents a manipulation scene as a sparse 3D skeleton composed of robot joints, object centers, and interaction points.
Constructed online from current RGB-D observations and robot proprioception, the skeleton provides a unified geometric state for action generation and future skeleton prediction.
Future skeleton prediction provides additional geometric supervision for action learning without requiring visual reconstruction.

Method

Given a language instruction $\ell$, the current RGB-D observation $o_{t}$, and the robot proprioceptive state $q_{t}$, our objective is to generate an action chunk where $\mathbf{a}_{t+i}\in\mathbb{R}^{d_{a}}$ denotes a single robot action, $H_{a}$ is the action horizon, and $d_{a}$ is the action dimension.

SkeleWAM represents the current scene as a sparse 3D skeleton, where $N$ is the number of skeleton nodes and $\Phi_{\psi}$ is the skeleton extractor described in Section 3.2 .

Method — SkeleWAM: Skeleton World-Action Modeling for Efficient Robotic Manipulation — Figure 2 : Overview of SkeleWAM. Frozen visual perception and forward kinematics construct the current skeleton. Action generation and future skeleton prediction are jointly trained with shared current-skeleton context. At inference, future skeleton prediction is omitted, and Medoid Action Consensus (MAC) selects a representative sampled action chunk. The robot executes its first $H_{e}$ actions before observing and replanning.

Experiments

We evaluate SkeleWAM on the full LIBERO-Plus benchmark [ 6 ] , which contains 10,030 variants across seven perturbation categories. We report zero-shot task success rates without fine-tuning on the evaluation variants. Table 1 compares SkeleWAM with VLA policies and world-model-based methods.

We compare SkeleWAM with representative VLA policies, including OpenVLA, OpenVLA-OFT, NORA, UniVLA, $\pi_{0}$, $\pi_{0}$-Fast, and $\pi_{0.5}$.

Experiments — SkeleWAM: Skeleton World-Action Modeling for Efficient Robotic Manipulation — Figure 3 : Qualitative real-world results. Representative rollouts for opening a drawer, placing a block in a drawer, and stacking bowls. Each row shows the early, interaction, and late stages of a successful rollout, together with the corresponding sparse 3D skeleton. Success rates are computed over 20 trials per task.

Sources and Demonstrations

Figures are reproduced from Sheng et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.