Robotics Research Hub
FERPO: Forward Entropy-Regularized Policy Optimization

FERPO: Forward Entropy-Regularized Policy Optimization

Sebastian Sanokowski, Alireza Sarmadi, Majid Khadiv · N/A · 2026

reinforcement-learning control benchmark
PDF arXiv Code (GitHub)

Framework

N/A

License

N/A

Stars

N/A

Summary

Several state-of-the-art methods for online reinforcement learning in continuous control improve policies using action gradients of a learned critic.

Abstract Summary

Several state-of-the-art methods for online reinforcement learning in continuous control improve policies using action gradients of a learned critic. However, critics are typically trained to predict returns, and accurate value predictions do not necessarily yield accurate action derivatives, potentially leading to unreliable policy updates. We propose Forward Entropy-Regularized Policy Optimization (FERPO), an on-policy maximum entropy reinforcement learning algorithm that performs policy improvement using critic values without differentiating the critic with respect to actions. FERPO derives an optimal target action distribution from a policy-improvement objective regularized by entropy and Kullback-Leibler (KL) divergence. We then fit the actor to this target by minimizing a forward-KL objective, estimated using self-normalized importance sampling (SNIS) with actions drawn from the rollout policy. By limiting the target distribution's deviation from the rollout policy, the KL regularization helps keep these importance weights well behaved. In contrast to reverse-KL objectives, which can favor a subset of the target distribution's modes, the forward-KL objective encourages coverage of multiple high-value modes and thereby promotes exploration. Experiments and ablations on MuJoCo Playground and ManiSkill show competitive performance and sample-efficiency gains. Computational benchmarks also demonstrate faster actor updates than Relative Entropy Pathwise Policy Optimization (REPPO).

Key Points

  • Several state-of-the-art methods for online reinforcement learning in continuous control improve policies using action gradients of a learned critic.
  • However, critics are typically trained to predict returns, and accurate value predictions do not necessarily yield accurate action derivatives, potentially leading to unreliable policy updates.
  • We propose Forward Entropy-Regularized Policy Optimization (FERPO), an on-policy maximum entropy reinforcement learning algorithm that performs policy improvement using critic values without differentiating the critic with respect to actions.
  • FERPO derives an optimal target action distribution from a policy-improvement objective regularized by entropy and Kullback-Leibler (KL) divergence.
  • We then fit the actor to this target by minimizing a forward-KL objective, estimated using self-normalized importance sampling (SNIS) with actions drawn from the rollout policy.

Problem Description

Maximum-entropy reinforcement learning. We consider a discounted Markov decision process with continuous state space $\mathcal{S}$, bounded continuous action space $\mathcal{A}$, initial-state density $\rho_{0}$, transition density $P(s^{\prime}\mid s,a)$, reward $r(s,a)$, and discount factor $\gamma\in[0,1)$.

For a policy $\pi$, let $\rho_{t}^{\pi}$ denote the state distribution at time $t$, generated by $s_{0}\sim\rho_{0}$, $a_{t}\sim\pi(\cdot\mid s_{t})$, and $s_{t+1}\sim P(\cdot\mid s_{t},a_{t})$.

Problem Description — FERPO: Forward Entropy-Regularized Policy Optimization
Figure 1: Actor directions and local losses. Top: green/red arrows point toward/away from the cost minimum (star); lengths are illustrative, not convergence guarantees. Bottom: frozen-reference losses at A–C, zeroed at the reference; KL losses are scaled by $T$. Details: Appendix A.1 .

Method: FERPO

Pathwise actor updates require accurate critic action derivatives, although standard critic learning supervises values. Value accuracy alone does not ensure derivative accuracy ( ?? ). This motivates an actor update that uses critic values without requiring $\nabla_{a}Q_{\phi}(s,a)$. We introduce Forward Entropy-Regularized Policy Optimization ( FERPO ). FERPO first minimizes the Lagrangian in Eq.

Method: FERPO — FERPO: Forward Entropy-Regularized Policy Optimization
Figure 2: Update cost on an A100. Medians and ranges over three repetitions with 131,072 rollout states. Actor time includes cache construction; total updates include critic training. PyTorch allocation excludes simulation and unused reservations.

Experiments

We compare actor updates on the bimodal cost $c(a)=(a^{2}-1)^{2}-0.1a$, whose right well is better. We use an exact critic $Q=-c$ and Gaussian policies $\pi_{\mu}=\mathcal{N}(\mu,\sigma^{2})$, with fixed $\sigma=0.4$, $\tau=0.1$, $\lambda=0.05$, and $T=\tau+\lambda$. Forward fitting changes the update direction. Let $\nu$ denote the candidate mean and $m_{*}:=\mathbb{E}_{q_{\mu}^{*}}[a]$.

We differentiate with respect to $\nu$, holding the reference $\pi_{\mu}$ and target $q_{\mu}^{*}$ fixed.

Experiments — FERPO: Forward Entropy-Regularized Policy Optimization
Figure 3: Ablations. Left: G1 candidate count and temperature at 50.07M steps (eight seeds for $K=2,16$ and $\tau=0$; ten for $K=4,8$). Middle: cached versus uncached G1 updates ($K=8$). Right: state-value versus $Q$ critics on AcrobotSwingupSparse (sparse rewards, $K=16$). Means with one SEM (left, middle) or one SD (right); panels use separate controls. The state-value variant uses $17\times$ more simulator transitions per rollout step.

Sources and Demonstrations

Figures are reproduced from Sanokowski et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.

Share

Related Papers

A Sim-to-Real Integration Pipeline for Training and Deployment of Chunk-Based VLA Manipulation Policies
arXiv preprint

A Sim-to-Real Integration Pipeline for Training and Deployment of Chunk-Based VLA Manipulation Policies

Mathilde Kappel, Clémence Grislain, Mohamed Chetouani et al.

Vision-Language-Action (VLA) models have become a prominent paradigm for mapping multimodal inputs, including semantic instructions, visual observations of the scene, and proprioceptive observations, to robot actions.

manipulation vision vla reinforcement-learning sim-to-real control learning-from-demonstration benchmark
PDF Advanced