When Plans Change, Remember: Replanning-Guided Visual Memory for Vision-Language-Action Models

Victoria Smelova, Daniil Kazachkov, Georgiy Kazantsev, Iuliia Tsvetkova, Egor Cherepanov, Nikita Kachaev, Artem Latyshev, Aleksandr I. Panov, Alexey K. Kovalev · N/A · 2026

Summary

Abstract Summary

Long-horizon, multi-stage Vision-Language-Action (VLA) policies often require visual information that is no longer present in the current observation. Retaining the complete observation history is computationally expensive and highly redundant, motivating selective visual memory. We introduce RePlanMem-VLA (Replanning-Guided Keyframe Memory for VLA), which turns a policy's own plan revisions into an online memory-write signal. Under receding-horizon execution, consecutive action chunks predict some of the same future actions. RePlanMem-VLA treats unusually large disagreement between these temporally aligned predictions as evidence of a task-relevant event. We first compare 35 disagreement measures across single-task autoregressive decoder-only and Flow-Matching Transformers trained from scratch, then validate the selected signals on pretrained OpenVLA-OFT and $\pi_{0.5}$ policies over MIKASA-Robo-VLA tasks. The resulting detectors achieve ROC-AUC of 0.981-0.997. We then use the selected signals in RePlanMem-VLA to trigger updates to a bounded visual memory, with a robot-masked write filter suppressing redundant frames. In the evaluated memory-retention experiments, RePlanMem-VLA preserves task-relevant observations until the later decision point in 100% of episodes. Real-robot replay data further show that event selection and redundancy suppression extend beyond simulation. These results establish plan revision as a policy-internal signal for controlling what a VLA remembers. Website: https://vika787.github.io/RePlanMem/.

Key Points

Long-horizon, multi-stage Vision-Language-Action (VLA) policies often require visual information that is no longer present in the current observation.
Retaining the complete observation history is computationally expensive and highly redundant, motivating selective visual memory.
We introduce RePlanMem-VLA (Replanning-Guided Keyframe Memory for VLA), which turns a policy's own plan revisions into an online memory-write signal.
Under receding-horizon execution, consecutive action chunks predict some of the same future actions
RePlanMem-VLA treats unusually large disagreement between these temporally aligned predictions as evidence of a task-relevant event.

RePlanMem-VLA

In long-horizon manipulation, task-relevant visual information often leaves the cameras before it is needed. A policy that sees only the current frame cannot recover that cue, so we concatenate the live views with a bounded visual memory of $B$ slots. Slots are empty at the start of an episode and are never carried across episode boundaries.

The policy predicts a chunk of $K$ actions from one observation and executes only the first $H 1 and 4 and Algorithm 1 summarize the procedure.

RePlanMem-VLA — When Plans Change, Remember: Replanning-Guided Visual Memory for Vision-Language-Action Models — Figure 2 : Real-robot delayed-cue setup. The robot observes the target object’s color. The cue then leaves the observation, and the robot must later select the matching object among distractors using stored visual information.

Experimental evaluation

Does revision of overlapping action chunks contain a task-relevant signal, and which disagreement measures extract it most reliably? Does this signal transfer to pretrained VLA policies across different backbones and replanning frequencies, and can it support accurate online event detection? Can the resulting surprise signal be used to drive selective visual-memory updates during VLA fine-tuning?

We first examine whether disagreement between overlapping action predictions contains task-relevant information using single-task autoregressive and F...

Experimental evaluation — When Plans Change, Remember: Replanning-Guided Visual Memory for Vision-Language-Action Models — Figure 3 : Surprise-gated memory updates on the RememberColor task. The surprise score $S_{q}$ and task-calibrated percentile $p_{q}$ identify candidate update times near transitions between the cue, empty, and action phases.

Sources and Demonstrations

Figures are reproduced from Smelova et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.