RoboMP-DINOv2: Prompts, Not Filters for Robust Robot Manipulation

Summary

Robot manipulation policies must generalize across visual shifts while preserving scene context relevant to action.

Abstract Summary

Robot manipulation policies must generalize across visual shifts while preserving scene context relevant to action. General-purpose vision encoders are not tailored to visuomotor control, while object-centric approaches often use segmentation masks as hard filters that discard potentially useful context. We propose RoboMP-DINOv2 (Robotics Mask-Prompted DINOv2), a full-scene vision encoder that treats masks as spatial prompts rather than visibility filters. It extracts dense DINOv2 features from the full observation, injects learned region-specific embeddings at masked locations, and jointly contextualizes prompted and unprompted tokens for action prediction. We further introduce masked-region color randomization (MCR) to improve appearance robustness, yielding RoboMP-DINOv2-MCR. Across seven simulated manipulation settings, RoboMP-DINOv2 achieves 60.7% success under spatial shifts and 59.7% under scene clutter, compared with 50.7% and 41.0% for a DINOv2-based Diffusion Policy. Under unseen object colors, RoboMP-DINOv2-MCR achieves 72.5% success versus 35.1% for the strongest color-randomized baseline. Additional experiments and representation analyses show improved robustness while preserving behaviorally relevant scene information. Code is available at https://github.com/han20192019/RoboMP_DINOv2.

Key Points

Robot manipulation policies must generalize across visual shifts while preserving scene context relevant to action.
General-purpose vision encoders are not tailored to visuomotor control, while object-centric approaches often use segmentation masks as hard filters that discard potentially useful context.
We propose RoboMP-DINOv2 (Robotics Mask-Prompted DINOv2), a full-scene vision encoder that treats masks as spatial prompts rather than visibility filters.
It extracts dense DINOv2 features from the full observation, injects learned region-specific embeddings at masked locations, and jointly contextualizes prompted and unprompted tokens for action prediction.
We further introduce masked-region color randomization (MCR) to improve appearance robustness, yielding RoboMP-DINOv2-MCR.

Method

Our goal is to incorporate region-level spatial guidance into a pretrained vision encoder without restricting the policy to a predefined set of segmented regions. The proposed RoboMP-DINOv2 (Robotics Mask-Prompted DINOv2) encoder comprises four stages: dense full-scene DINOv2 feature extraction, feature-space mask prompting, global contextualization, and scene-level aggregation.

RoboMP-DINOv2 first extracts dense visual features from the complete RGB observation and then injects learned prompts at locations specified by the input masks.

Method — RoboMP-DINOv2: Prompts, Not Filters for Robust Robot Manipulation — Figure 1: Seven simulated manipulation settings under the three evaluation regimes. Columns denote task settings, and rows show representative observations from Clean Spatial OOD, Clutter OOD, and Appearance OOD.

Experiments

Our experiments evaluate the proposed approach under three forms of visual distribution shift. We first evaluate RoboMP-DINOv2 under held-out spatial configurations and added scene clutter. We then evaluate the RoboMP-DINOv2-MCR training pipeline under unseen task-object appearances.

Finally, we compare RoboMP-DINOv2 with closely matched architectural variants and analyze how strongly the learned representations respond to the evaluated visual perturbations. We consider seven simulated manipulation settings (Fig.

Experiments — RoboMP-DINOv2: Prompts, Not Filters for Robust Robot Manipulation — Figure 3: Preserving behaviorally relevant color cues outside the randomized regions. The stove-light color determines the correct action. RoboMP-DINOv2-MCR retains access to this contextual signal and substantially outperforms DP–DINOv2 trained with full-image color randomization under both light conditions.

Sources and Demonstrations

Figures are reproduced from Qi et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.