Memory as Plans: World-Action Modeling with Memory-Grounded Planning
Sizhe Zhao, Haozhe Xie, Weiyu Zhao, Chenchu Zhang, Huan Wang, Chenyang Wang, Qinglin Liu, Shengping Zhang · N/A · 2026
Framework
N/A
License
N/A
Stars
N/A
Summary
Mainstream robotic policies often adopt a Markovian formulation, but many complex real-world manipulation tasks are inherently non-Markovian, requiring long-horizon memory beyond the current observation. Existing memory mechanisms often rely on language summaries, growing visual windows, or their...
Abstract Summary
Key Points
- Mainstream robotic policies often adopt a Markovian formulation, but many complex real-world mani...
- Existing memory mechanisms often rely on language summaries, growing visual windows, or their com...
- We introduce MaP-WAM, a Memory-as-Plans framework that decomposes memory-dependent world-action m...
- MaP-WAM represents memory as completed segment records containing language instructions and spars...
- A World-Action-Progress (WAP) model executes each plan over an unknown duration by jointly predic...
Memory as Plans: World-Action Modeling with Memory-Grounded Planning
|Authors: Sizhe Zhao, Haozhe Xie, Weiyu Zhao, Chenchu Zhang, Huan Wang, Chenyang Wang, Qinglin Liu, Shengping Zhang
|Venue: arXiv preprint | Year: 2026
|arXiv: 2609.11561v1
Abstract
Mainstream robotic policies often adopt a Markovian formulation, but many complex real-world manipulation tasks are inherently non-Markovian, requiring long-horizon memory beyond the current observation. Existing memory mechanisms often rely on language summaries, growing visual windows, or their combinations, and may therefore lose fine-grained visual evidence or face a trade-off between history coverage and execution efficiency. We introduce MaP-WAM, a Memory-as-Plans framework that decomposes memory-dependent world-action modeling into memory-grounded planning and plan-conditioned execution, and uses long-term multimodal episodic context as planning-time evidence rather than repeatedly conditioning the executor on the full history. MaP-WAM represents memory as completed segment records containing language instructions and sparse visual context, and converts this episodic memory into compact plans comprising the next segment-level language plan and corresponding visual guidance. A World-Action-Progress (WAP) model executes each plan over an unknown duration by jointly predicting action chunks and corresponding execution progress at inference time, calibrating predicted progress through plan-observation alignment for adaptive segment transitions and closed-loop context updates. MaP-WAM keeps the executor context length fixed, while structured attention further enables key-value caching in both planning and execution. MaP-WAM achieves state-of-the-art performance on RMBench with an 83.3% success rate and attains 78.0% success on real-robot tasks, while maintaining approximately constant executor inference latency as task history grows.
Key Contributions
- Mainstream robotic policies often adopt a Markovian formulation, but many complex real-world mani…
- Existing memory mechanisms often rely on language summaries, growing visual windows, or their com…
- We introduce MaP-WAM, a Memory-as-Plans framework that decomposes memory-dependent world-action m…
- MaP-WAM represents memory as completed segment records containing language instructions and spars…
- A World-Action-Progress (WAP) model executes each plan over an unknown duration by jointly predic…
Topics
- manipulation
- vision
- reinforcement-learning
- planning
Code & Data
No code repository linked in paper metadata.
BibTeX
@article{Zhao2026_260911561v1,
title = {Memory as Plans: World-Action Modeling with Memory-Grounded Planning},
author = {Sizhe Zhao and Haozhe Xie and Weiyu Zhao and Chenchu Zhang and Huan Wang and Chenyang Wang and Qinglin Liu and Shengping Zhang},
year = {2026},
eprint = {2609.11561v1},
archivePrefix = {arXiv},
primaryClass = {cs.RO},
url = {https://arxiv.org/abs/2609.11561v1}
}
Related Papers
Evidence-Gated Task and Motion Planning with Vision-Language Models
Tsunehiko Tanaka, Matthew Stephenson, Alistair Macvicar et al. · arXiv preprint · Aug 2026
Robots executing long-horizon manipulation tasks from natural-language instructions must reason about both semantic task structure and geometric feasibility. However, under partial observability, the availability of goal-relevant objects may be uncertain. In such cases, approaches that combine Vi...
3DThinkVLA: Endowing Vision-Language-Action Models with Latent 3D Priors via 3D-Thinking-Guided Co-training
Jiaxin Shi, Xidong Zhang, Fucai Zhu et al. · arXiv preprint · Jun 2026
We propose a 3D-thinking-guided co-training framework that enables vision-language-action (VLA) models to perform 3D spatial reasoning implicitly during action prediction. Our core insight is that 3D geometry perception and 3D spatial reasoning are distinct capabilities that can be disentangled and ...
A Low-Cost, Open Platform for End-to-End Autonomous Driving on a Miniature Ackermann Vehicle
Gustavo Claudio Karl Couto, Eric Aislan Antonelo, Gabriel George Zipperer · arXiv preprint · Sep 2026
This paper presents a low-cost, open experimental platform for research in end-to-end autonomous driving with miniature Ackermann vehicles. The platform combines a physical vehicle, a printed urban track, data collection tools, trajectory registration, and a Webots digital twin, enabling controll...
Adaptive Vision-Language Grasping via Composable Foundation Priors and Generalizable Grasp Synthesis
Sixu Yan, Shikang Wang, Binhua Huang et al. · arXiv preprint · Sep 2026
This paper proposes AdaRoboVLG, a task-adaptive Vision-Language-Grasp (VLG) framework that supports generalizable grasp synthesis across different robotic hands. Unlike existing VLG methods that tightly couple foundation models with end-to-end grasp policies, AdaRoboVLG learns an efficient genera...