Robotics Research Hub
UniMPA: A Unified Memory-Prediction-Action Model via Action-Grounded Transition Modeling

UniMPA: A Unified Memory-Prediction-Action Model via Action-Grounded Transition Modeling

Wei Li, Rui Shao, Jie He, Lingsen Zhang, Ziwei Liu, Liqiang Nie · N/A · 2026

manipulation vision vla reinforcement-learning
PDF arXiv Code (GitHub)

Framework

N/A

License

N/A

Stars

N/A

Summary

Recent advances in Vision-Language-Action (VLA) models have improved robotic manipulation, yet observation-to-action learning remains limited by a fundamental transition realizability gap, manifested in three tightly coupled problems: (i) Transition ambiguity.

Abstract Summary

Recent advances in Vision-Language-Action (VLA) models have improved robotic manipulation, yet observation-to-action learning remains limited by a fundamental transition realizability gap, manifested in three tightly coupled problems: (i) Transition ambiguity. Visually similar current observations may correspond to different manipulation phases and imply different subsequent transitions. (ii) Prediction--execution mismatch. A visually plausible predicted future observation does not necessarily correspond to a physically realizable transition. (iii) Experience--realization mismatch. A historically executable action pattern may not necessarily realize the intended transition in the current scene and therefore requires context-aware adaptation. Accordingly, we propose UniMPA, a Unified Memory-Prediction-Action model that addresses these problems through a shared action-grounded transition interface. (i) UniMPA introduces Persistent-Selective Future Prediction to resolve transition ambiguity by modeling the intended future state evolution. A persistent latent stream continuously tracks task-level progress, while a transition-critical pixel stream selectively resolves fine-grained interaction changes through memory-grounded prediction. (ii) To assess the physical executability of the anticipated transition, the predicted transition queries a temporal Visual-Action Memory Bank. The bank retrieves historically realized visual-action experience, grounding future prediction in executable evidence. (iii) To adapt executable experience to the current scene, an Action-Visual Memory Bank retrieves visually grounded action prototypes from historical action evolution. Prototype-Biased Flow then shifts the flow source toward a historically supported action manifold for context-aware refinement.

Key Points

  • Recent advances in Vision-Language-Action (VLA) models have improved robotic manipulation, yet observation-to-action learning remains limited by a fundamental transition realizability gap, manifested in three tightly coupled problems: (i) Transition ambiguity.
  • Visually similar current observations may correspond to different manipulation phases and imply different subsequent transitions.
  • (ii) Prediction--execution mismatch
  • A visually plausible predicted future observation does not necessarily correspond to a physically realizable transition.
  • (iii) Experience--realization mismatch

UniMPA

Given a robot manipulation trajectory, UniMPA takes as input the current multi-view observation, language instruction, robot proprioceptive state, and historical action context.

Formally, at timestep $t$, we define the input context as where $\mathcal{V}_{t}=\{\mathbf{I}^{\nu}_{t}\}_{\nu\in\mathcal{U}}$ contains the RGB observations from the camera set $\mathcal{U}$, $\mathbf{q}_{t}$ denotes the robot proprioceptive state, $\bm{\ell}$ is the language instruction, and $\mathbf{H}_{t}\in\mathbb{R}^{M\times D}$ denotes the historical action window...

UniMPA — UniMPA: A Unified Memory-Prediction-Action Model via Action-Grounded Transition Modeling
Fig. 4: Transition-Critical Trigger Gate. Top: The gate combines latent semantic variation with action-derived translation, rotation, and gripper-state indicators. Pixel-level future prediction is activated when any criterion indicates a critical transition. Bottom: Visualization of transition-critical observation locations on representative trajectories. The highlighted timestamps show frames selected by the Trigger Gate. The selected frames cluster around interaction-intensive moments, such as reaching, contact, pose adjustment, and gripper switching, showing that the trigger gate concentrates pixel-level supervision on task-relevant transitions while avoiding redundant prediction.

Experiments

We evaluate UniMPA on four simulation benchmarks covering zero-shot robustness, cross-setting generalization, and semantic reasoning (Fig. 6 Left). LIBERO [ 96 ] measures manipulation performance across four suites (Spatial, Object, Goal, and Long) which respectively emphasize spatial relationships, object-centric transfer, goal-conditioned behavior, and long-horizon composition.

Experiments — UniMPA: A Unified Memory-Prediction-Action Model via Action-Grounded Transition Modeling
Fig. 5: Action-manifold prototype refinement. Historical action evolution retrieves a visually grounded action prototype from the Action-Visual Memory Bank. The retrieved prototype is fused with recent action history to form an action-manifold prior, which translates the Gaussian source distribution toward a historically supported motion region. The Action Expert then refines the prior-biased samples through flow matching according to the current vision-language context and future-supervised latent transition representation.

Sources and Demonstrations

Figures are reproduced from Li et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.

Share

Related Papers

A Sim-to-Real Integration Pipeline for Training and Deployment of Chunk-Based VLA Manipulation Policies
arXiv preprint

A Sim-to-Real Integration Pipeline for Training and Deployment of Chunk-Based VLA Manipulation Policies

Mathilde Kappel, Clémence Grislain, Mohamed Chetouani et al.

Vision-Language-Action (VLA) models have become a prominent paradigm for mapping multimodal inputs, including semantic instructions, visual observations of the scene, and proprioceptive observations, to robot actions.

manipulation vision vla reinforcement-learning sim-to-real control learning-from-demonstration benchmark
PDF Advanced
Beyond Reconstruction: What Matters in Action Tokenization for Robot Policies?
arXiv preprint

Beyond Reconstruction: What Matters in Action Tokenization for Robot Policies?

Haoran Chen, Jingtian Ji, Samuel Wheeler et al.

Autoregressive action-token policies such as vision-language-action models require action tokenizers to translate discrete token sequences into precise control actions in continuous space. Many action tokenizers learn the mapping between tokens and actions via a reconstruction objective.

manipulation vision vla reinforcement-learning control benchmark
PDF Intermediate