Robotics Research Hub
Sample-Efficient Diffusion-based Reinforcement Learning with Critic Guidance

Sample-Efficient Diffusion-based Reinforcement Learning with Critic Guidance

Featured

Shutong Ding, Zejia Zhong, Zhongyi Wang, Ke Hu, Bikang Pan, Jingya Wang, Ye Shi · · 2026

diffusion-policy reinforcement-learning locomotion manipulation
PDF arXiv

Framework

N/A

License

N/A

Stars

0

Summary

Recent advances in reinforcement learning (RL) have achieved great successes by leveraging the multimodality and exploration capability of diffusion policies. Among these approaches, one representativ

Abstract Summary

Recent advances in reinforcement learning (RL) have achieved great successes by leveraging the multimodality and exploration capability of diffusion policies. Among these approaches, one representative branch focuses on the sampling-based policy optimization. This design enables better exploration capability of the diffusion model, particularly at the beginning of training, but suffer from low exploitation in Q-value information, resulting in a slow policy convergence. Another branch pays attention to gradient-based policy optimization, which sufficiently exploits the gradient of the Q function yet tends to collapse into a unimodal policy with low diversity. To address this issue, we propose CGPO, Critic-Guided diffusion Policy Optimization, which effectively balances exploration and exploitation with the training-free guidance technique integrated into the denoising process of diffusion policy. Concretely, CGPO steers action generation toward high-value regions defined by the critic network and uses the guided actions as regression objectives. In this manner, CGPO reduces the time required to obtain high-quality actions and improves final performance with better balance between the exploration-exploitation tradeoff. We validate the effectiveness of CGPO on 5 MuJoCo locomotion tasks, and CGPO achieves state-of-the-art performance compared with existing diffusion-based RL methods. Notably, CGPO is the first success to incorporate diffusion policy into real-world RL, with its superior performance on Franka robot arm grasping tasks. Our official page is released at https://dingsht.tech/cgpo-webpage.

Key Points

  • Proposes CGPO, a critic-guided diffusion policy optimization method.
  • Balances exploration and exploitation using training-free guidance during denoising.
  • Achieves SOTA on 5 MuJoCo locomotion tasks.
  • First diffusion policy applied to real-world RL on Franka robot grasping.
  • Uses guided actions as regression objectives for faster policy convergence.

Preliminaries

Reinforcement learning (RL) studies sequential decision-making under uncertainty, where an agent interacts with an environment to maximize long-term return.

A discounted Markov decision process (MDP) is defined as $\mathcal{M}=(\mathcal{S},\mathcal{A},P,r,\gamma)$, where $\mathcal{S}$ and $\mathcal{A}\subseteq\mathbb{R}^{d}$ denote the state and continuous action spaces, $P(\cdot\mid s,a)$ is the transition kernel, $r:\mathcal{S}\times\mathcal{A}\to\mathbb{R}$ is the reward function, and $\gamma\in(0,1)$ is the discount factor.

Preliminaries — Sample-Efficient Diffusion-based Reinforcement Learning with Critic Guidance
Figure 2 : t-SNE visualization of sampled candidate actions on HalfCheetah-v3 at different training stages (left to right). Points are colored by critic-based labels (good vs. bad). As the policy concentrates, candidate diversity shrinks, and within-set separability decreases, consistent with a reduced critic contrast under finite candidate sampling.

Methods

In this section, we present Critic-Guided Diffusion Policy Optimization (CGPO). Our motivation is that sampling-based diffusion-policy improvement becomes less effective as training proceeds: under a fixed sampling budget, action candidates drawn from a progressively concentrated policy tend to be similar, making critic scores less discriminative and weakening the practical update signal ( Ding et al., 2024 ) .

CGPO replaces multi-candidate selection with a single guided target per state.

Methods — Sample-Efficient Diffusion-based Reinforcement Learning with Critic Guidance
Figure 4 : Sequential video frames of the real-world evaluation tasks. The top row shows the cube stacking task, and the bottom row shows the cylindrical peg-in-hole task.

Experiments

We evaluate CGPO in simulation and on a real robot to assess both learning performance and practical deployability.

Our experiments are designed to answer three questions: (i) does CGPO improve online diffusion-policy reinforcement learning under a fixed interaction budget; (ii) does training-time critic-guided target synthesis yield more reliable policy improvement than sampling-based candidate selection when the policy distribution becomes concentrated; and (iii) can these gains be obtained without increasing deployment-time computation, by k...

Experiments — Sample-Efficient Diffusion-based Reinforcement Learning with Critic Guidance
Figure 6 : Experimental setup with the Franka Emika Panda robot and Robotiq 2F-85 gripper.

mpact Statement

This work advances diffusion-based reinforcement learning by introducing CGPO, which utilize critic guidance that improves the exploration–exploitation balance and enables diffusion policies to be trained directly through interaction.

By reducing reliance on large-scale human demonstrations in imitation learning, CGPO has the potential to facilitate more practical and scalable deployment of diffusion policies in real-world robotic systems.

Sources and Demonstrations

Figures are reproduced from Ding et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.

Share

Related Papers

Accelerating and Scaling MPC-Guided Reinforcement Learning for Humanoid Locomotion and Manipulation
arXiv preprint

Accelerating and Scaling MPC-Guided Reinforcement Learning for Humanoid Locomotion and Manipulation

Junheng Li, Liang Wu, Sergio A. Esteban et al.

In humanoid motion control, model predictive control (MPC) offers physically grounded prediction and constraint handling, while reinforcement learning (RL) enables robust whole-body skills through large-scale simulation. However, using MPC inside RL often requires time-consuming problem construction or excessive training overhead, making such frameworks difficult to justify in practice. This work studies efficient training-time MPC guidance for humanoid locomotion and manipulation, termed MPC-RL. We introduce a centroidal-dynamics MPC reward formulation that leverages guidance from MPC trajectories in training time. To make this practical in massively parallel RL, we develop π^nMPC, a parallel-in-horizon and construction-free batched GPU MPC solver that operates directly on time-varying dynamics to avoid high memory usage and pre-compilation. Through a variety of comparative studies and hardware validations, we have found that MPC-RL achieves superior performance in locomotion and manipulation skills.

humanoid reinforcement-learning model-predictive-control locomotion manipulation
Code PDF Advanced
GitHub ★ —
An Analysis of Streaming Deep Reinforcement Learning for Adaptive Continual Learning in Robotics
arXiv preprint

An Analysis of Streaming Deep Reinforcement Learning for Adaptive Continual Learning in Robotics

Teeratham Vitchutripop, Alyssa Quarles, Wenhe Zhang et al.

Over the course of a lifetime, robots may encounter novel scenarios unaccounted for in its original training that result in performance degradation. One common approach to mitigating this issue is to further grow the offline training dataset in hopes of producing a policy robust to these changes.

manipulation locomotion reinforcement-learning benchmark
PDF Intermediate