Memory as Plans: World-Action Modeling with Memory-Grounded Planning

Sizhe Zhao, Haozhe Xie, Weiyu Zhao, Chenchu Zhang, Huan Wang, Chenyang Wang, Qinglin Liu, Shengping Zhang · N/A · 2026

Summary

Abstract Summary

Mainstream robotic policies often adopt a Markovian formulation, but many complex real-world manipulation tasks are inherently non-Markovian, requiring long-horizon memory beyond the current observation. Existing memory mechanisms often rely on language summaries, growing visual windows, or their combinations, and may therefore lose fine-grained visual evidence or face a trade-off between history coverage and execution efficiency. We introduce MaP-WAM, a Memory-as-Plans framework that decomposes memory-dependent world-action modeling into memory-grounded planning and plan-conditioned execution, and uses long-term multimodal episodic context as planning-time evidence rather than repeatedly conditioning the executor on the full history. MaP-WAM represents memory as completed segment records containing language instructions and sparse visual context, and converts this episodic memory into compact plans comprising the next segment-level language plan and corresponding visual guidance. A World-Action-Progress (WAP) model executes each plan over an unknown duration by jointly predicting action chunks and corresponding execution progress at inference time, calibrating predicted progress through plan-observation alignment for adaptive segment transitions and closed-loop context updates. MaP-WAM keeps the executor context length fixed, while structured attention further enables key-value caching in both planning and execution. MaP-WAM achieves state-of-the-art performance on RMBench with an 83.3% success rate and attains 78.0% success on real-robot tasks, while maintaining approximately constant executor inference latency as task history grows.

Key Points

Mainstream robotic policies often adopt a Markovian formulation, but many complex real-world manipulation tasks are inherently non-Markovian, requiring long-horizon memory beyond the current observation.
Existing memory mechanisms often rely on language summaries, growing visual windows, or their combinations, and may therefore lose fine-grained visual evidence or face a trade-off between history coverage and execution efficiency.
We introduce MaP-WAM, a Memory-as-Plans framework that decomposes memory-dependent world-action modeling into memory-grounded planning and plan-conditioned execution, and uses long-term multimodal episodic context as planning-time evidence rather than repeatedly conditioning the executor on the full history.
MaP-WAM represents memory as completed segment records containing language instructions and sparse visual context, and converts this episodic memory into compact plans comprising the next segment-level language plan and corresponding visual guidance.
A World-Action-Progress (WAP) model executes each plan over an unknown duration by jointly predicting action chunks and corresponding execution progress at inference time, calibrating predicted progress through plan-observation alignment for adaptive segment transitions and closed-loop context updates.

Our Approach

Problem Formulation. We formulate memory-dependent robotic manipulation as a sequential decision-making problem. Given a language instruction l , the current proprioceptive state $\text{s}_{t}$, and the observation sequence $\text{f}_{\leq t}$, a general memory-dependent policy models where $\text{a}_{t+1:t+h}$ denotes the next action chunk.

The key challenge is that critical information for action generation may reside in historical observations $\text{f}_{

Our Approach — Memory as Plans: World-Action Modeling with Memory-Grounded Planning — Figure 3 : Real-world tasks and success rates. We report the success rates over 50 trials per task.

Experiments

Model Configuration. For language planning, we fine-tune Qwen3.5-4B ( Qwen Team, 2026 ) to predict the next segment-level language plan from the history. For visual planning, we initialize the CWM from WAN-2.2-5B ( Wang et al., 2025 ) and fine-tune it using a causal input format and a block-causal attention mask. Each completed segment is uniformly resampled into $N=8$ frames as sparse visual context.

Sources and Demonstrations

Figures are reproduced from Zhao et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.