CFG-Policy: Unified Diffusion Policy Learning Framework via Classifier-free Guidance

Shutong Ding, Zhaoyang Liu, Yimiao Zhou, Jingya Wang, Jiayuan Gu, Ye Shi · N/A · 2026

Summary

Abstract Summary

Diffusion policies are increasingly used in robotic control and manipulation, vision-language-action models, and world-model-based agents. Their ability to represent multimodal action distributions offers rich opportunities for exploration in reinforcement learning (RL). However, optimizing diffusion policies with RL can reduce the behavioral diversity and further limit their exploration capability. In RL post-training, directly fine-tuning a pretrained policy may also disrupt its acquired capabilities, while action-space residuals may lack the expressive power for multimodal exploration. Together, these issues reveal a common challenge across online and offline-to-online diffusion RL: improving task performance while retaining the behavioral diversity and capabilities needed for continued adaptation. We introduce CFG-Policy, a unified, plug-and-play framework that separates exploration and exploitation into two generative policies and combines their score or velocity fields during sampling through classifier-free guidance. For online RL, a frozen exploration branch sustains behavioral diversity while an exploitation branch learns from task rewards; for offline-to-online RL, a pretrained exploitation branch remains frozen while a trainable branch steers the composed behavior. CFG-Policy preserves the underlying RL objectives and improves performance with QSM and QVPO on 5 MuJoCo tasks and with Flow-SDE+PPO on 4 RoboTwin manipulation tasks. Project page is at https://zhy-liu.github.io/cfg-policy-page.

Key Points

Diffusion policies are increasingly used in robotic control and manipulation, vision-language-action models, and world-model-based agents.
Their ability to represent multimodal action distributions offers rich opportunities for exploration in reinforcement learning (RL).
However, optimizing diffusion policies with RL can reduce the behavioral diversity and further limit their exploration capability.
In RL post-training, directly fine-tuning a pretrained policy may also disrupt its acquired capabilities, while action-space residuals may lack the expressive power for multimodal exploration.
Together, these issues reveal a common challenge across online and offline-to-online diffusion RL: improving task performance while retaining the behavioral diversity and capabilities needed for continued adaptation.

Method

As shown in Figure 2 , we propose CFG-Policy which aims to address the two learning regimes discussed in the Introduction through a common principle: preserve one behavioral capability in a frozen generative policy while optimizing a complementary one .

The two policies are composed during diffusion or flow sampling, allowing the same mechanism to preserve exploration in online RL and pretrained behavior in RL post-training. Consider a diffusion or flow policy that generates an action $a=x_{0}$ from $x_{T}\sim\mathcal{N}(0,I)$.

Method — CFG-Policy: Unified Diffusion Policy Learning Framework via Classifier-free Guidance — Figure 1: Exploration–exploitation challenges in diffusion RL. In online diffusion RL, optimizing a single generative policy can progressively reduce behavioral diversity and miss better modes. In VLA RL post-training, directly fine-tuning the pretrained policy may disrupt pretrained capabilities, while residual policies can provide limited exploration.

Experiments

We evaluate CFG-Policy in the two complementary regimes introduced in the Method: online RL and RL post-training.

Our experiments study whether (i) CFG-Policy can be applied to different generative RL algorithms without changing their original objectives, (ii) the same compositional mechanism is effective in both learning regimes, and (iii) adaptive generative composition is preferable to fixed guidance or simple action-space blending.

We evaluate on five MuJoCo tasks: HalfCheetah-v4, Walker2d-v4, Ant-v4, Humanoid-v4, and Hopper-v4.

Experiments — CFG-Policy: Unified Diffusion Policy Learning Framework via Classifier-free Guidance — Figure 2: The sampling process of CFG-Policy. It composes two generative policies with different behavioral preferences in score space with guidance weight $w$.

Sources and Demonstrations

Figures are reproduced from Ding et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.