Robotics Research Hub
Dual Advantage Fields

Dual Advantage Fields

Alexey Zemtsov, Maxim Bobrin, Alexander Nikulin, Dmitry V. Dylov, Fakhri Karray, Vladislav Kurenkov, Martin Takáč, Arip Asadulaev · N/A · 2026

reinforcement-learning manipulation locomotion
PDF arXiv

Framework

N/A

License

N/A

Stars

N/A

Summary

Offline goal-conditioned reinforcement learning requires both long-horizon reachability estimates and local action comparisons. Dual goal representations provide value fields that capture global goal reachability, but they do not directly specify which action should be preferred at a given state.

Abstract Summary

Offline goal-conditioned reinforcement learning requires both long-horizon reachability estimates and local action comparisons. Dual goal representations provide value fields that capture global goal reachability, but they do not directly specify which action should be preferred at a given state. We propose Dual Advantage Fields, a policy-extraction method that turns a bilinear dual value model into a local advantage signal. Under bilinear dual parameterization, the goal embedding is the gradient of the value field with respect to the state representation. DAF learns an action-effect model that predicts the discounted feature displacement induced by an action and scores actions by the alignment between this displacement and the goal direction. In the realizable case, this score equals the goal-conditioned Bellman advantage, yielding a standard local policy-improvement guarantee. On OGBench locomotion, manipulation, and puzzle tasks, DAF improves aggregate RLiable metrics and performs strongly in settings where locally correct actions differ from direct movement toward the final goal.

Key Points

  • Improves locomotion stability and agility
  • Addresses dexterous manipulation challenges
  • Incorporates tactile or force feedback for robust interaction
  • Learns multimodal motion skills from human demonstrations
  • Theoretically studies value/advantage decomposition for RL

Preliminaries

We study offline goal-conditioned reinforcement learning (GCRL) [ 14 , 5 , 15 , 17 ] : the learner has access to a fixed offline dataset of transitions but cannot collect new experience in the environment [ 19 ] . The objective is to infer a an optimal goal-conditioned policy even for unseen during training combinations of state-goal pairs.

Let $\mathcal{S}$ and $\mathcal{A}$ denote state and action spaces, and let $\mathcal{G}\subseteq\mathcal{S}$ (or an abstract goal space) denote goals.

Preliminaries — Dual Advantage Fields
Figure 2 : Dual Advantage Fields. Under a bilinear goal-conditioned value model, the goal embedding defines a direction in representation space. DAF scores an action by projecting its induced feature displacement onto this goal direction, yielding a local advantage-like signal for policy improvement.

Dual Advantage Fields

Our method is based on a simple insight from bilinear value decomposition in Eq. ( 5 ).

Holding the goal fixed and viewing the value as a function of the state embedding $\psi$, we have Under the bilinear goal-conditioned value model $V_{\theta}(s,g)=\psi_{\theta}(s)^{\top}\phi_{\theta}(g)$, the gradient of the value with respect to the state embedding is the goal embedding: Thus, the goal embedding $\phi_{\theta}(g)$ is the value-gradient direction in representation space under the Euclidean geometry of the learned embedding.

Dual Advantage Fields — Dual Advantage Fields
Figure 3 : Pre-grasp vector field in cube-single . Arrows show decoded high-level directions from sampled gripper positions around the cube, with the cube and final goal fixed. DAF points locally toward the cube before grasping, while OTA points toward the terminal placement goal. The yellow marker denotes the mean decoded target.

Training and goal-conditioned policy extraction

Dayan and Singh [4] showed that policy improvement can be organized around relative measures of how actions compare at a state- merits that need not reduce to a fully trusted global value oracle.

In the goal-conditioned setting, the Bellman advantage $A^{\pi}(s,a,g)$ in ( 8 ) is exactly such an object: it ranks $a$ by the expected one-step gain in return, isolating the effect of the transition from the baseline $V^{\pi}(s,g)$. Our bilinear potential ( 5 ) turns this comparison into explicit geometry in $\psi$.

Experiments

In this section, we empirically validate the findings developed in the previous sections on the OGBench benchmark [ 19 ] . OGBench is designed to evaluate several core capabilities required by offline goal-conditioned reinforcement learning, including long-horizon reasoning, trajectory stitching, generalization to unseen goals, robustness to suboptimal data, and control under imperfect offline coverage.

Sources and Demonstrations

Figures are reproduced from Zemtsov et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.

Share

Related Papers

Accelerating and Scaling MPC-Guided Reinforcement Learning for Humanoid Locomotion and Manipulation
arXiv preprint

Accelerating and Scaling MPC-Guided Reinforcement Learning for Humanoid Locomotion and Manipulation

Junheng Li, Liang Wu, Sergio A. Esteban et al.

In humanoid motion control, model predictive control (MPC) offers physically grounded prediction and constraint handling, while reinforcement learning (RL) enables robust whole-body skills through large-scale simulation. However, using MPC inside RL often requires time-consuming problem construction or excessive training overhead, making such frameworks difficult to justify in practice. This work studies efficient training-time MPC guidance for humanoid locomotion and manipulation, termed MPC-RL. We introduce a centroidal-dynamics MPC reward formulation that leverages guidance from MPC trajectories in training time. To make this practical in massively parallel RL, we develop π^nMPC, a parallel-in-horizon and construction-free batched GPU MPC solver that operates directly on time-varying dynamics to avoid high memory usage and pre-compilation. Through a variety of comparative studies and hardware validations, we have found that MPC-RL achieves superior performance in locomotion and manipulation skills.

humanoid reinforcement-learning model-predictive-control locomotion manipulation
Code PDF Advanced
GitHub ★ —
An Analysis of Streaming Deep Reinforcement Learning for Adaptive Continual Learning in Robotics
arXiv preprint

An Analysis of Streaming Deep Reinforcement Learning for Adaptive Continual Learning in Robotics

Teeratham Vitchutripop, Alyssa Quarles, Wenhe Zhang et al.

Over the course of a lifetime, robots may encounter novel scenarios unaccounted for in its original training that result in performance degradation. One common approach to mitigating this issue is to further grow the offline training dataset in hopes of producing a policy robust to these changes.

manipulation locomotion reinforcement-learning benchmark
PDF Intermediate