JAMB: Joint Action-Motion Diffusion for Bimanual Manipulation

Summary

Coordinated bimanual manipulation is challenging because the motion of either arm can alter the shared 3D scene and thereby affect the other arm.

Abstract Summary

Coordinated bimanual manipulation is challenging because the motion of either arm can alter the shared 3D scene and thereby affect the other arm. Yet most diffusion policies generate actions without explicitly modeling these future geometric consequences, while predictive variants typically use future state only as auxiliary supervision or fixed conditioning. We address this limitation by proposing JAMB, a diffusion policy that jointly denoises bimanual actions and future 3D point tracks. By allowing action and track hypotheses to evolve together within a shared Transformer, each can inform and refine the other throughout denoising. We further ground multimodal representations in a shared spatiotemporal coordinate system to facilitate geometry-aware interaction during joint denoising. We evaluate JAMB on diverse bimanual manipulation tasks in RoboTwin 2.0 and on a real-world robot, comparing it with action-only policies and alternative future-prediction approaches spanning different state representations and learning objectives. Across 16 simulation tasks, JAMB achieves an average success rate of 83.4%, outperforming the strongest baseline by 23.9 percentage points. On three real-world tasks, it outperforms the action-only and auxiliary geometry prediction methods by 50.0 and 21.2 percentage points, respectively. Beyond these performance gains, JAMB shows stronger generalization to cluttered scenes and out-of-distribution backgrounds than the evaluated baselines. Together, these results demonstrate the effectiveness of our joint action-motion modeling framework for coordinated bimanual manipulation. Our project website is available at https://jam-bimanual.github.io/

Key Points

Coordinated bimanual manipulation is challenging because the motion of either arm can alter the shared 3D scene and thereby affect the other arm.
Yet most diffusion policies generate actions without explicitly modeling these future geometric consequences, while predictive variants typically use future state only as auxiliary supervision or fixed conditioning.
We address this limitation by proposing JAMB, a diffusion policy that jointly denoises bimanual actions and future 3D point tracks.
By allowing action and track hypotheses to evolve together within a shared Transformer, each can inform and refine the other throughout denoising.
We further ground multimodal representations in a shared spatiotemporal coordinate system to facilitate geometry-aware interaction during joint denoising.

PROBLEM FORMULATION

We assume a dataset of expert trajectories where $o_{t}$ denotes the visual observation, $s_{t}$ denotes the robot proprioceptive state, and $a_{t}\in\mathbb{R}^{D_{a}}$ denotes the bimanual robot action. At time step $t$, the policy predicts an action chunk To represent future scene motion, we associate the visual observation with a grid of $N$ patch-aligned 3D query points.

Let $q_{i,0}\in\mathbb{R}^{3}$ denote the current position of query $i$, and let $q_{i,h}$ denote its position at future step $h$.

PROBLEM FORMULATION — JAMB: Joint Action-Motion Diffusion for Bimanual Manipulation — Fig. 3: Real-world bimanual setup with two xArm manipulators, a fixed ZED Mini stereo camera, and a Meta VR headset for teleoperation. We evaluate on three tasks covering complementary forms of bimanual coordination.

Method

We construct patch-aligned 3D point tracks as an explicit representation of future scene motion. For each visual patch, we initialize one image-space query at the patch center, yielding a fixed grid of $N$ points distributed over the input image. Each query is then lifted into 3D using the corresponding depth and camera geometry, producing a set of scene points whose motion is tracked over the future horizon.

Experiments

Does predicting future 3D point tracks improve bimanual policy performance? Q2: How do joint action–motion diffusion and spatiotemporal grounding affect policy performance? Q3: Does our method generalize better to unseen backgrounds and object appearances? We evaluate on RoboTwin 2.0 [ 16 ] , following the two bimanual task categories used in prior work: 8 Sync-bimanual tasks, and 8 Seq-coordinate tasks.

Sync-bimanual tasks require simultaneous coordination, and Seq-coordinate tasks require sequential interaction between the two arms.

Experiments — JAMB: Joint Action-Motion Diffusion for Bimanual Manipulation — Fig. 5: Qualitative visualization of future 3D point tracks predicted by our method on three real-world tasks: Store Block (top), Stack Basin (middle), and Place Duck Box (bottom). Predicted future 3D point tracks are color-coded by time, progressing from red to yellow.

Sources and Demonstrations

Figures are reproduced from Xiao et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.