Value as Memory: Online Value-Guided Post-training for General Robot Policies

Wenye Yu, Yijin Chen, Kaiyi Xiong, Wenbo Han, Wendi Chen, Yang Jin, Han Xue, Jun Lv, Cewu Lu, Chuan Wen · N/A · 2026

Summary

Abstract Summary

Real-world post-training of generalist robot policies faces two challenges: partial observability makes action selection ambiguous, while offline post-training methods delay feedback from deployment experience. We present Value as Memory (VaM), a framework that addresses these challenges through value caching and online real-world reinforcement learning (RL). Value caching conditions actions on recent value predictions as task-progress memory, while online updates enable timely policy improvement by combining offline demonstrations, off-policy interventions, and on-policy rollouts for sample-efficient adaptation. As a result, VaM transitions from offline RL warmup to online autonomous learning with sparse rewards. Across long-horizon tasks involving deformable objects and contact-rich manipulation, VaM yields a mean absolute success-rate gain of $29.4\%$ with value caching, while achieving a $32.8\%$ gain with online autonomous learning over matched offline training. The evaluated online schedule performs $8\times$ as many policy iterations under the same gradient-step budget, with more than $2\times$ lower recorded time per gradient step with concurrent data collection and training.

Key Points

Real-world post-training of generalist robot policies faces two challenges: partial observability makes action selection ambiguous, while offline post-training methods delay feedback from deployment experience.
We present Value as Memory (VaM), a framework that addresses these challenges through value caching and online real-world reinforcement learning (RL).
Value caching conditions actions on recent value predictions as task-progress memory, while online updates enable timely policy improvement by combining offline demonstrations, off-policy interventions, and on-policy rollouts for sample-efficient adaptation.
As a result, VaM transitions from offline RL warmup to online autonomous learning with sparse rewards.
Across long-horizon tasks involving deformable objects and contact-rich manipulation, VaM yields a mean absolute success-rate gain of $29.4\%$ with value caching, while achieving a $32.8\%$ gain with online autonomous learning over matched offline training.

Preliminaries

We model iterative robot post-training as a Markov Decision Process (MDP) $\mathcal{M}=(\mathcal{S},\mathcal{A},P,r,\gamma)$, where $\mathcal{S}$ and $\mathcal{A}$ are the state and action spaces, $P(s_{t+1}\mid s_{t},a_{t})$ is the transition kernel, $r(s_{t},a_{t})$ is the reward function, and $\gamma\in[0,1)$ is the discount factor.

The policy $\pi_{\theta}(a_{t:t+H}\mid s_{t})$ has parameters $\theta$ and predicts the action sequence $a_{t:t+H}=(a_{t},\ldots,a_{t+H-1})$, where $H$ is the prediction horizon.

Preliminaries — Value as Memory: Online Value-Guided Post-training for General Robot Policies — Figure 1: VaM uses online real-world RL and value caching to boost sample efficiency and mitigate partial observability in long-horizon, deformable-object, and contact-rich manipulation.

Method

VaM uses a distributional value expert to supply recent progress predictions to the action expert through a value cache and advantage labels for policy post-training. Training proceeds through four stages: offline demonstration initialization, offline RL warmup, Online DAgger, and Learn from Experience (LfE). The value and action experts are updated in parallel during online RL.

Figure 2 summarizes this training pipeline and the value caching architecture.

Method — Value as Memory: Online Value-Guided Post-training for General Robot Policies — Figure 2: VaM training pipeline and model architecture. The top panel summarizes the four-stage offline-to-online training pipeline. The bottom panel shows how the distributional value expert turns previously computed value predictions into the value cache that conditions the action expert.

Experiments

How much does value caching improve policy performance in RL post-training? Why does value caching improve policy performance? How much does VaM’s offline-to-online RL system enhance sample efficiency? We consider three real-world manipulation tasks, illustrated in Fig. 3 : Box Assembly , Laundry Folding , and Magnetic Bus Assembly .

All three tasks are long-horizon manipulation workflows that cover contact-rich assembly and deformable-object manipulation. Appendix A.1 provides the detailed procedure and success criterion for each task.

Experiments — Value as Memory: Online Value-Guided Post-training for General Robot Policies — Figure 3: Task illustrations for Box Assembly (top), Laundry Folding (middle), and Magnetic Bus Assembly (bottom).

Sources and Demonstrations

Figures are reproduced from Yu et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.