Dual Advantage Fields
Alexey Zemtsov, Maxim Bobrin, Alexander Nikulin, Dmitry V. Dylov, Fakhri Karray, Vladislav Kurenkov, Martin Takáč, Arip Asadulaev · N/A · 2026
Framework
N/A
License
N/A
Stars
N/A
Summary
Offline goal-conditioned reinforcement learning requires both long-horizon reachability estimates and local action comparisons. Dual goal representations provide value fields that capture global goal reachability, but they do not directly specify which action should be preferred at a given state. We...
Abstract Summary
Key Points
- Improves locomotion stability and agility
- Addresses dexterous manipulation challenges
- Incorporates tactile or force feedback for robust interaction
- Learns multimodal motion skills from human demonstrations
- Theoretically studies value/advantage decomposition for RL
Dual Advantage Fields
Authors: Alexey Zemtsov, Maxim Bobrin, Alexander Nikulin, Dmitry V. Dylov, Fakhri Karray, Vladislav Kurenkov, Martin Takáč, Arip Asadulaev
Venue: ICML 2026 | Year: 2026
arXiv: 2606.04188v1
Abstract
Offline goal-conditioned reinforcement learning requires both long-horizon reachability estimates and local action comparisons. Dual goal representations provide value fields that capture global goal reachability, but they do not directly specify which action should be preferred at a given state. We propose Dual Advantage Fields, a policy-extraction method that turns a bilinear dual value model into a local advantage signal. Under bilinear dual parameterization, the goal embedding is the gradient of the value field with respect to the state representation. DAF learns an action-effect model that predicts the discounted feature displacement induced by an action and scores actions by the alignment between this displacement and the goal direction. In the realizable case, this score equals the goal-conditioned Bellman advantage, yielding a standard local policy-improvement guarantee. On OGBench locomotion, manipulation, and puzzle tasks, DAF improves aggregate RLiable metrics and performs strongly in settings where locally correct actions differ from direct movement toward the final goal.
Key Contributions
- Improves locomotion stability and agility
- Addresses dexterous manipulation challenges
- Incorporates tactile or force feedback for robust interaction
- Learns multimodal motion skills from human demonstrations
- Theoretically studies value/advantage decomposition for RL
Topics
- reinforcement-learning
- manipulation
- locomotion
Code & Data
No code repository linked in paper metadata.
BibTeX
@article{Zemtsov2026_260604188v1,
title={Dual Advantage Fields},
author={Alexey Zemtsov and Maxim Bobrin and Alexander Nikulin and Dmitry V. Dylov and Fakhri Karray and Vladislav Kurenkov and Martin Takáč and Arip Asadulaev},
year={2026},
eprint={2606.04188v1},
archivePrefix={arXiv},
primaryClass={cs.LG},
url={https://arxiv.org/abs/2606.04188v1}
}
Related Papers
Accelerating and Scaling MPC-Guided Reinforcement Learning for Humanoid Locomotion and Manipulation
Junheng Li, Liang Wu, Sergio A. Esteban et al. · arXiv preprint · Jun 2026
In humanoid motion control, model predictive control (MPC) offers physically grounded prediction and constraint handling, while reinforcement learning (RL) enables robust whole-body skills through large-scale simulation. However, using MPC inside RL often requires time-consuming problem construction or excessive training overhead, making such frameworks difficult to justify in practice. This work studies efficient training-time MPC guidance for humanoid locomotion and manipulation, termed MPC-RL. We introduce a centroidal-dynamics MPC reward formulation that leverages guidance from MPC trajectories in training time. To make this practical in massively parallel RL, we develop π^nMPC, a parallel-in-horizon and construction-free batched GPU MPC solver that operates directly on time-varying dynamics to avoid high memory usage and pre-compilation. Through a variety of comparative studies and hardware validations, we have found that MPC-RL achieves superior performance in locomotion and manipulation skills.
DECOWAM: Decoupled Whole-Body World-Action Model for Legged Mobile Manipulation
Siyuan Ma, Boshi Zhang, Yutian Zhang et al. · arXiv preprint · Aug 2026
Mobile manipulation requires a robot to predict how locomotion and arm motion jointly alter future observations and control. Existing world-action models, developed largely for fixed-base platforms, do not explicitly distinguish camera ego-motion from base and arm actions. Here we introduce DECOW...
M3imic: Learning a Versatile Whole-Body Controller for Multimodal Motion Mimicking
Zuxing Lu, Ziang Zheng, Yao Lyu et al. · arXiv preprint · Jun 2026
Building a general-purpose whole-body controller is essential for enabling diverse motion capabilities in humanoid robots across a wide range of downstream tasks, including locomotion and loco-manipulation. Different tasks rely on distinct motion reference modalities: locomotion primarily depends on...
Sample-Efficient Diffusion-based Reinforcement Learning with Critic Guidance
Shutong Ding, Zejia Zhong, Zhongyi Wang et al. · ICML 2026 · May 2026
Recent advances in reinforcement learning (RL) have achieved great successes by leveraging the multimodality and exploration capability of diffusion policies. Among these approaches, one representativ