Robotics Research Hub
DynaFLIP: Rethinking Robotics Perception via Tri-Modal-Dynamics Guided Representation

DynaFLIP: Rethinking Robotics Perception via Tri-Modal-Dynamics Guided Representation

Featured

Jusuk Lee, Seungjae Lee, Jonghun Shin, Hoseong Jung, Sungha Kim, Daesol Cho, H. Jin Kim, Jia-Bin Huang, Furong Huang · · 2026

foundation-models perception manipulation representation-learning vla
PDF arXiv

Framework

N/A

License

N/A

Stars

0

Summary

Robot manipulation critically depends on perception that preserves the action-relevant aspects of a scene. Yet most robot learning pipelines are built upon visual encoders pre-trained for static recog

Abstract Summary

Robot manipulation critically depends on perception that preserves the action-relevant aspects of a scene. Yet most robot learning pipelines are built upon visual encoders pre-trained for static recognition or vision-language alignment, leaving motion understanding to downstream policies. We introduce DynaFLIP, a dynamics-aware multimodal pre-training framework that pushes motion understanding upstream into perception. We construct image-language-3D flow triplets from heterogeneous human and robot videos, and use these triplets as training-time supervision to shape an image-only encoder. Our key idea is to encourage the three modalities to span a small simplex volume in the shared hyperspherical space -- a smaller simplex volume indicating stronger alignment. To avoid the geometric ambiguity and trivial collapse of naive volume minimization, we combine simplex-volume minimization with a cosine regularizer and a contrastive objective. Our analyses show that DynaFLIP focuses on control-relevant regions critical for manipulation. The resulting dynamics-aware representations serve as reusable visual backbones and consistently outperform baselines across diverse downstream policies, including VLAs. We validate this across diverse simulation and real-world setups, with gains reaching +22.5% under out-of-distribution scenarios. Our results suggest that robot generalization improves when visual representations are trained to encode not just what is present, but how the world changes under action.

Key Points

  • Dynamics-aware multimodal pre-training framework for robotics perception.
  • Uses image-language-3D flow triplets for training-time supervision.
  • Simplex volume minimization in hyperspherical space aligns tri-modal representations.
  • Works as reusable visual backbone for VLAs and downstream policies.
  • +22.5% gains under OOD scenarios across diverse simulation and real-world setups.

Method

DynaFLIP shifts visual pre-training from static scene understanding to motion-induced state transitions. Section 2.1 introduces a simplex-guided multimodal alignment objective that aligns image transitions, language, and 3D flow into a shared embedding space while resolving two optimization pitfalls: geometric ambiguity and trivial collapse.

Section 2.2 then presents the auxiliary objectives—temporal contrastive and actor losses—that further strengthen dynamics-aware visual representations.

Method — DynaFLIP: Rethinking Robotics Perception via Tri-Modal-Dynamics Guided Representation
Figure 2 : Overview of DynaFLIP. Three modalities are encoded into embeddings in a shared hyperspherical space. The image encoder (initialized from DINOv2 and fully fine-tuned) produces per-frame features from $I_{t},I_{t+H}$ via CLS and mean-pooled patch tokens, which are then fused into $z_{I}$. A frozen T5 with a learnable adapter produces $z_{L}$ from the EOS token of $L$, and a 3D flow encoder produces $z_{F}$ from $F_{t:t+K}$. The alignment loss minimizes the area $A$ spanned by these embeddings, with auxiliary actor and temporal contrastive losses reinforcing dynamics-aware representations. Our pre-trained image encoder serves as a visual backbone for diverse downstream policies, with the language encoder optionally included for instruction-conditioned policies.

Experiments

In this section, we evaluate DynaFLIP through extensive experiments in both simulation and the real world. Through these experiments, we aim to answer the following questions: Does DynaFLIP learn dynamics-aware representations that preserve control-relevant information for manipulation? Do dynamics-aware representations improve downstream policy learning compared to strong baselines?

Can DynaFLIP improve real-world manipulation under both in-distribution and out-of-distribution settings? Which design choices in DynaFLIP are most critical to its...

Experiments — DynaFLIP: Rethinking Robotics Perception via Tri-Modal-Dynamics Guided Representation
Figure 3 : Two optimization pitfalls of naïve simplex-volume minimization. (a) Geometric ambiguity. A flat triangle has near-zero area even when one modality remains far from the other two. The cosine regularizer pulls selected modality pairs together, yielding a desired alignment (see Eq. ( 2 )). (b) Trivial collapse. Without negative tuples, all modality embeddings collapse to a single point. Negative tuples in our contrastive framework push apart mismatched configurations, preventing collapse (see Eq. ( 3 )).

Sources and Demonstrations

Figures are reproduced from Lee et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.

Share

Related Papers