Mem4Act: Learning Effective Latent Memory for Long-Horizon Robotic Manipulation

Yun Zhang, Zhiyu Huang, Tianhui Cai, Chen Tang, Wei Wang, Jiaqi Ma · N/A · 2026

Summary

Long-horizon robotic manipulation fundamentally requires memory: agents must retain task-relevant information from past observations and interactions and use it to guide future actions.

Abstract Summary

Long-horizon robotic manipulation fundamentally requires memory: agents must retain task-relevant information from past observations and interactions and use it to guide future actions. Existing approaches typically preserve raw historical observations, compressed visual tokens, or language summaries, but these representations either expose the action policy to increasingly long histories or risk discarding task-relevant temporal, semantic, and perceptual information. We introduce Mem4Act, a framework for learning effective latent memory as a fixed-size representation explicitly optimized for long-horizon robotic control. Mem4Act addresses three coupled challenges: how to efficiently cover long histories, what information memory should retain, and how to ensure the policy actually relies on memory. To this end, it organizes history via multi-granularity temporal aggregation, preserves complementary semantic and perceptual information through question-answering supervision and key-frame DINO feature reconstruction, and employs memory-first two-stage training to encourage greater use of memory by the action policy. The resulting memory captures both high-level task state and fine-grained perceptual evidence while providing a bounded interface to the action policy. Across RoboMME, RMBench, and four real-world manipulation tasks, Mem4Act achieves state-of-the-art performance on memory-dependent control, with success rates of 49.88%, 61.0%, and 81.5%, respectively. Extensive ablations further demonstrate the importance of temporal aggregation, complementary memory supervision, and memory-first training.

Key Points

Long-horizon robotic manipulation fundamentally requires memory: agents must retain task-relevant information from past observations and interactions and use it to guide future actions.
Existing approaches typically preserve raw historical observations, compressed visual tokens, or language summaries, but these representations either expose the action policy to increasingly long histories or risk discarding task-relevant temporal, semantic, and perceptual information.
We introduce Mem4Act, a framework for learning effective latent memory as a fixed-size representation explicitly optimized for long-horizon robotic control.
Mem4Act addresses three coupled challenges: how to efficiently cover long histories, what information memory should retain, and how to ensure the policy actually relies on memory.
To this end, it organizes history via multi-granularity temporal aggregation, preserves complementary semantic and perceptual information through question-answering supervision and key-frame DINO feature reconstruction, and employs memory-first two-stage training to encourage greater use of memory by the action policy.

Method

We consider partially observable, long-horizon robotic manipulation. At timestep $t$, the agent predicts a future action chunk $\mathcal{A}_{t}=(a_{t},\ldots,a_{t+T-1})$ given instruction $L$, current observation $O_{t}$, and history $\mathcal{H}_{t}=(O_{1},\ldots,O_{t})$.

Experiments

Datasets. We initialize Mem4Act from a pretrained $\pi_{0.5}$ model ( Physical Intelligence et al., 2025 ) and train it on two memory-intensive long-horizon manipulation benchmarks. RMBench ( Chen et al., 2026a ) contains 9 dual-arm tasks that require recalling either a single relevant past observation $\mathcal{M}(1)$ or multiple observations $\mathcal{M}(n)$ for successful completion.

RoboMME ( Dai et al., 2026 ) evaluates 16 single-arm tasks spanning temporal, spatial, object, and procedural memory. Baselines.

Experiments — Mem4Act: Learning Effective Latent Memory for Long-Horizon Robotic Manipulation — Figure 2: Mem4Act structure. (a) Historical observations are encoded as compact visual tokens with state, phase, and time embeddings. (b) The latent memory tokens first attend within fine temporal windows, then are regrouped to attend broader windows. Global self-attention produces the final latent memory $M_{t}$. (c) Memory-related question answering and multi-keyframe DINO feature reconstruction supervise what this memory retains. (d) The resulting memory conditions action generation through memory modulation in the action expert. (e) Training first grounds the memory with retrospective feature supervision, then jointly adapts memory and policy for action generation.

Sources and Demonstrations

Figures are reproduced from Zhang et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.