Diffusion Policy: Visuomotor Policy Learning via Action Diffusion

Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, Shuran Song · Columbia University, University of Tokyo, Toyota Research Institute · 2023

Summary

A behavior cloning approach that models robot policies as conditional diffusion processes, enabling multimodal action distributions and smooth action sequences.

Abstract Summary

Diffusion Policy rethinks behavior cloning by treating policy learning as a generative modeling problem. Rather than predicting a single deterministic action or fitting a unimodal Gaussian, the method trains a denoising diffusion model to generate a coherent action sequence conditioned on the current visual observation. Starting from random noise, the model iteratively refines its output over a series of denoising steps until a smooth, feasible action chunk emerges. This formulation elegantly solves a long-standing problem in imitation learning: how to represent multimodal action distributions without suffering from averaging artifacts. In traditional behavior cloning, when the demonstration dataset contains multiple valid strategies for the same scenario—such as pushing an object from the left versus the right—a deterministic policy tends to average these options into a meaningless compromise. Diffusion Policy avoids this by learning the full distribution of demonstrated actions. Because diffusion models are inherently multimodal, they can capture diverse valid strategies and sample from them at test time. Additionally, the model predicts short action horizons (e.g., 8 or 16 future timesteps) rather than isolated instantaneous actions. This chunk-based prediction leads to significantly smoother and more temporally consistent robot behavior, reducing jerk and mechanical wear. The architecture is surprisingly flexible. The authors show that the denoising network can be implemented as a convolutional U-Net, a transformer, or even a simple MLP, depending on the observation and action space. Training follows the standard conditional diffusion objective: given an observation, the model learns to reverse the noising process applied to the corresponding action chunk from the demonstration dataset. At inference time, the policy runs a fixed number of denoising iterations (typically 10 to 100) to generate the final action sequence. The paper also demonstrates that inference acceleration techniques such as DDIM can cut computational cost with minimal impact on task success. Empirical evaluation spans both simulated benchmarks and real hardware. On the classic Push-T contact-rich manipulation task, Diffusion Policy outperforms prior state-of-the-art methods including ACT and VINN by a wide margin. It also achieves strong results on the Franka Kitchen multi-task benchmark and a suite of real-world robot tasks such as pouring and slot insertion. The hardware experiments confirm that the smooth, multimodal actions generated by diffusion translate directly into higher success rates and more robust behavior on physical robots.

Key Points

Formulates behavior cloning as conditional diffusion over action sequences.
Handles multimodal action distributions, avoiding deterministic averaging artifacts.
Predicts action chunks (short horizons) for temporal smoothness and consistency.
Compatible with CNN, Transformer, or MLP denoising backbones.
Strong results on Push-T, Franka Kitchen, and real robot manipulation tasks.

Diffusion Policy Formulation

We formulate visuomotor robot policies as Denoising Diffusion Probabilistic Models (DDPMs) [ 21 ] . Crucially, Diffusion policies are able to express complex multimodal action distributions and possess stable training behavior – requiring little task-specific hyperparameter tuning. The following sections describe DDPMs in more detail and explain how they may be adapted to represent visuomotor policies.

Key Design Decisions

In this section, we describe key design decisions for Diffusion Policy as well as its concrete implementation of $\epsilon_{\theta}$ with neural network architectures. The first design decision is the choice of neural network architectures for $\epsilon_{\theta}$. In this work, we examine two common network architecture types, convolutional neural networks (CNNs) Ronneberger et al.

(2015) and Transformers Vaswani et al. (2017) , and compare their performance and training characteristics.

Key Design Decisions — Diffusion Policy: Visuomotor Policy Learning via Action Diffusion — Figure 10: Realworld Sauce Manipulation. [Left] 6DoF pouring Task. The robot needs to \raisebox{-0.9pt}{1}⃝ dip the ladle to scoop sauce from the bowl, \raisebox{-0.9pt}{2}⃝ approach the center of the pizza dough, \raisebox{-0.9pt}{3}⃝ pour sauce, and \raisebox{-0.9pt}{4}⃝ lift the ladle to finish the task. [Right] Periodic spreading Task The robot needs to \raisebox{-0.9pt}{1}⃝ approach the center of the sauce with a grasped spoon, \raisebox{-0.9pt}{2}⃝ spread the sauce to cover pizza in a spiral pattern, and \raisebox{-0.9pt}{3}⃝ lift the spoon to finish the task.

Intriguing Properties of Diffusion Policy

In this section, we provide some insights and intuitions about diffusion policy and its advantages over other forms of policy representations. The challenge of modeling multi-modal distribution in human demonstrations has been widely discussed in behavior cloning literature Florence et al. (2021) ; Shafiullah et al. (2022) ; Mandlekar et al. (2021) .

Diffusion Policy’s ability to express multimodal distributions naturally and precisely is one of its key advantages.

Intriguing Properties of Diffusion Policy — Diffusion Policy: Visuomotor Policy Learning via Action Diffusion — Figure 11 : Bimanual Egg Beater Manipulation. The robot needs to \raisebox{-0.9pt}{1}⃝ push the bowl into position (only if too close to the left arm), \raisebox{-0.9pt}{2}⃝ approach and pick up the egg beater with the right arm, \raisebox{-0.9pt}{3}⃝ place the egg beater in the bowl, \raisebox{-0.9pt}{4}⃝ approach and grasp the egg beater crank handle, and \raisebox{-0.9pt}{5}⃝ turn the crank handle 3 or more times.

Evaluation

We systematically evaluate Diffusion Policy on 15 tasks from 4 benchmarks Florence et al. (2021) ; Gupta et al. (2019) ; Mandlekar et al. (2021) ; Shafiullah et al. (2022) . This evaluation suite includes both simulated and real environments, single and multiple task benchmarks, fully actuated and under-actuated systems, and rigid and fluid objects.

We found Diffusion Policy to consistently outperform the prior state-of-the-art on all of the tested benchmarks, with an average success-rate improvement of 46.9%.

Evaluation — Diffusion Policy: Visuomotor Policy Learning via Action Diffusion — Figure 12 : Bimanual Mat Unrolling. The robot needs to \raisebox{-0.9pt}{1}⃝ pick up one side of the mat (if needed), using the left or right arm, \raisebox{-0.9pt}{2}⃝ lift and unroll the mat (if needed), \raisebox{-0.9pt}{3}⃝ ensure that both sides of the mat are grasped, \raisebox{-0.9pt}{4}⃝ lift the mat, \raisebox{-0.9pt}{5}⃝ place the mat oriented with the table, mostly centered, and \raisebox{-0.9pt}{6}⃝ release the mat.

Realworld Evaluation

We evaluated Diffusion Policy in the realworld performance on 4 tasks across 2 hardware setups – with training data from different demonstrators for each setup. On the realworld Push-T task, we perform ablations examining Diffusion Policy on 2 architecture options and 3 visual encoder options; we also benchmarked against 2 baseline methods with both position-control and velocity-control action spaces.

On all tasks, Diffusion Policy variants with both CNN backbones and end-to-end-trained visual encoders yielded the best performance.

Realworld Evaluation — Diffusion Policy: Visuomotor Policy Learning via Action Diffusion — Figure 13 : Bimanual Shirt Folding. The robot needs to \raisebox{-0.9pt}{1}⃝ approach and grasp the closest sleeve with both arms, \raisebox{-0.9pt}{2}⃝ fold the sleeve and release, \raisebox{-0.9pt}{3}⃝ drag the shirt closer (if needed), \raisebox{-0.9pt}{4}⃝ approach and grasp the other sleeve with both arms, \raisebox{-0.9pt}{5}⃝ fold the sleeve and release, \raisebox{-0.9pt}{6}⃝ drag the shirt to a orientation for folding, \raisebox{-0.9pt}{7}⃝ grasp and fold the shirt in half by its collar, \raisebox{-0.9pt}{8}⃝ drag the shirt to the center, and \raisebox{-0.9pt}{9}⃝ smooth out the shirt and move the arms away.

Realworld Bimanual Tasks

Beyond single arm setup, we further demonstrate Diffusion Policy on several challenging bimanual tasks. To enable bimanual tasks, the majority of effort was spent on extending our robot stack to support multi-arm teleopration and control. Diffusion Policy worked out of the box for these tasks without hyperparameter tuning.

The proprioceptive observation space is extended to include the poses of both end-effectors and the gripper widths of both grippers.

Sources and Demonstrations

Figures are reproduced from Chi et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.