Robotics Research Hub
Grasp, Handover, Rotate: Bimanual Object Reorientation via Compositional Diffusion and Energy-Based Optimization

Grasp, Handover, Rotate: Bimanual Object Reorientation via Compositional Diffusion and Energy-Based Optimization

Wun Lam Yeung, Wenjun Liu, Yui Cheung Yu, Zhengyan Lambo Qin, Qijin She, Heng Li, Ziqi Wang, Ping Tan · N/A · 2026

manipulation reinforcement-learning sim-to-real planning
PDF arXiv

Framework

N/A

License

N/A

Stars

N/A

Summary

Bimanual object reorientation - picking an object, handing it over between two arms, and placing it in a desired target pose - is valuable when direct placement from the initial grasp is infeasible due to collisions, kinematic constraints, or poor final orientation.

Abstract Summary

Bimanual object reorientation - picking an object, handing it over between two arms, and placing it in a desired target pose - is valuable when direct placement from the initial grasp is infeasible due to collisions, kinematic constraints, or poor final orientation. However, achieving this under multiple competing objectives remains challenging. We introduce BiCompoDiff, a compositional diffusion and energy-based framework that jointly optimizes grasp selection, handover, regrasp, and motion planning under multiple constraints. By combining a pretrained grasp diffusion model with bimanual planning energy-based models (EBMs), our method injects gradient guidance during reverse diffusion to enforce collision avoidance, trajectory smoothness (via differentiable inverse kinematics), handover feasibility, and regrasp safety. Annealed MCMC sampling further refines grasp poses over the composite energy landscape. Experiments across diverse simulated household reorientation tasks demonstrate that BiCompoDiff achieves over 20% higher success rates and up to 37% smoother trajectories (measured by joint displacement) compared to strong sampling-based baselines. Real-world validation confirms effective sim-to-real transfer and robust performance on challenging scenes.

Key Points

  • Bimanual object reorientation - picking an object, handing it over between two arms, and placing it in a desired target pose - is valuable when direct placement from the initial grasp is infeasible due to collisions, kinematic constraints, or poor final orientation.
  • However, achieving this under multiple competing objectives remains challenging
  • We introduce BiCompoDiff, a compositional diffusion and energy-based framework that jointly optimizes grasp selection, handover, regrasp, and motion planning under multiple constraints.
  • By combining a pretrained grasp diffusion model with bimanual planning energy-based models (EBMs), our method injects gradient guidance during reverse diffusion to enforce collision avoidance, trajectory smoothness (via differentiable inverse kinematics), handover feasibility, and regrasp safety.
  • Annealed MCMC sampling further refines grasp poses over the composite energy landscape

PRELIMINARIES

In this work, we leverage diffusion models as the core generative component for grasp candidate initialization and subsequent joint refinement with motion planning. Diffusion models are particularly well-suited for this setup due to their close connection to energy-based models (EBMs) and their ability to support compositional optimization in high-dimensional, multi-modal spaces [ 6 , 20 ] .

PRELIMINARIES — Grasp, Handover, Rotate: Bimanual Object Reorientation via Compositional Diffusion and Energy-Based Optimization
Fig. 3 : Overview of the proposed BiCompoDiff framework pipeline. Phase 1: Noisy initial poses ($g_{PK}^{t}$, $g_{PL}^{t}$, $g_{HO}^{t}$ in world frame) are randomly sampled, and $g_{RG}^{t}$ is derived from the other three. Point clouds (object + environment) are fed into the grasp diffusion model and the bimanual planning EBMs. Phase 2: The diffusion model predicts $\epsilon$-scores with current timestep $t$ and noisy poses ($g_{PK}^{t}$, $g_{PL}^{t}$ converted to object frame), which are combined with energy gradients from the compositional bimanual planning EBMs (encoding smoothness, regrasp, handover, and grasp collision costs). These guidance signals iteratively refine the noisy poses ($g_{PK}^{t}$, $g_{PL}^{t}$, $g_{HO}^{t}$) at each reverse diffusion timestep. Phase 3: At timestep $t=0$, collision-free pose pairs are filtered and ranked by composite scores. The highest-scoring pair is then passed to the sandbox module to generate the whole bimanual reorientation trajectory.

METHODS

We study coordinated dual-arm pick-and-place in clutter using two UR12e 6-DoF robots. The robot must optimize a full reorientation sequence: pick with one arm, handover, regrasp with the other arm, and place at the target.

METHODS — Grasp, Handover, Rotate: Bimanual Object Reorientation via Compositional Diffusion and Energy-Based Optimization
Fig. 4 : Visualization of bimanual reorientation trajectories for placing a bridge-shape brick from lying flat on the ground to standing upright on a shelf. BiCompoDiff-Full minimizes joint displacement from PK to HO while enabling smooth side-regrasping for final placement. BiCompoDiff-HO optimizes handover to a near-standing pose, reducing reorientation effort during the final RG to PL phase and yielding the most efficient RG to PL trajectory, but at the cost of higher joint delta earlier. NoEBM produces inefficient trajectories, requiring excessive object flipping and rear regrasp. Overall, Full mode achieves the best balance between PK to HO and RG to PL trajectory efficiency.

Experiments

We evaluate BiCompoDiff on collaborative bimanual manipulation tasks involving pick, place, handover, and regrasp actions for object reorientation. All experiments are conducted in simulation unless otherwise stated, with real-world validation. Experiments are conducted in a modular simulation pipeline that couples a high-level task specification front-end with a motion planning back-end.

The environment includes: Task Sandbox: Defines scene geometry, initial/target object poses, and planner interface.

Sources and Demonstrations

Figures are reproduced from Yeung et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.

Share

Related Papers

Dexterity from Supervised Play
arXiv preprint

Dexterity from Supervised Play

Botao He, Yiming Dou, Yiannis Aloimonos et al.

How can robots autonomously collect the contact-rich experience needed to learn dexterous manipulation?

manipulation reinforcement-learning sim-to-real planning control learning-from-demonstration benchmark
Code PDF Intermediate
GitHub ★ —