See What I Need, Dream What I Want: A Navigation Foundation Model on a Self-Constructed Representation Space

At a glance

Problem: navigation policies built on vision foundation models mostly see what those encoders' pretraining objectives chose to keep. Source
Idea: attend over layers of semantic, geometric and dynamic encoders to build the policy's own space, then roll out latent predictions of future observations before outputting waypoints. Source
Evidence: on the VLNVerse test split it reaches 83.80% success with fine-grained instructions, 20.05 points above Qwen-RobotNav-8B. Source

Evaluation scope The component ablations use one point-goal protocol on 4,000 trajectories where success is near its ceiling, so differences show mainly in collision rates. Source

How it works

DINOv3, Depth Anything 3 and the OpenD4RT video encoder stay frozen; light adapters align features from layers spread across each encoder's depth. Source
A Qwen3-VL backbone predicts eight waypoints with position and yaw through one output shared by point-goal navigation, tracking and vision-and-language navigation. Source
After the visual and instruction context, the language model generates four continuous latents, each fed back as the next input, and a projector maps them to the target space of future observations. Source
Because the prediction targets come from the same trainable Representation Residuals, VISReg is applied to them to keep the space from collapsing. Source

RepresentNav overview: Representation Residuals attend over layers of three frozen vision foundation models, and a latent chain-of-thought predicts future tokens in that space before the planning head acts. Original figure

Evidence & results

On EVT-Bench distracted tracking with a single camera it reaches an 85.5 tracking rate, 10.1 points above the best monocular baseline, although TrackVLA and TrackVLA++ keep lower collision rates. Evidence
In point-goal ablations, building the constructed space from the three encoders reaches 93.45% success and 18.05% collisions, versus 87.30% and 28.70% when their outputs are concatenated. Evidence
Predicting the future in the self-constructed space lowers collisions from 18.05% to 13.45%, while predicting the backbone ViT's features raises them to 22.00%. Evidence

Real-world point-goal navigation, person following and vision-and-language navigation, with occupancy, target-box and semantic readouts in the insets. Original figure

Limitations & boundaries

The construction has been validated only on navigation; manipulation, dexterous hands and humanoid whole-body control remain untested. Source
On VLN-CE RxR, Qwen-RobotNav-4B still leads the other metrics besides the tied nDTW. Source
Real-world behavior is shown qualitatively in the main text; the quantitative tests on a Go2 quadruped and a Bumi humanoid are in the appendix. Source

Takeaway

Our reading: the ablations suggest that both choices matter, letting the policy construct the space it sees through and predicting the future inside that same space, but the evidence is limited to navigation. Source

Resources

Full text ↗PDF ↗

Figures: the paper's authors. Editorial source and image review: 2026-10-26. Findings are author-reported, not independently reproduced by Robotics Research Hub.

Original abstract

What visual representation should a navigation policy act on? Navigation policies built on vision foundation models largely leave representation design to the encoders' pretraining objectives. We present RepresentNav, a navigation foundation model built on a self-constructed representation space: a visual representation constructed by the policy's own objectives from vision foundation models. RepresentNav builds this space with Representation Residuals, a compact three-layer transformer that attends over features spanning the full depth of multiple complementary vision foundation models, so that the policy learns to see what it needs for navigation. In the same space it dreams what it wants: before every action a latent chain-of-thought predicts the representation of the scene a moment ahead. The dream is trained end-to-end with the policy as a joint-embedding predictive architecture (JEPA), with a next-representation prediction loss and a regularizer on its targets. RepresentNav sets a new state of the art on VLNVerse, VLN-CE R2R, EVT-Bench, and two point-goal benchmarks, and its gains carry over to a physical robot amid clutter and moving people. On VLNVerse fine-grained, RepresentNav achieves 83.80% success, outperforming the previous state of the art by 20.05 percentage points, and on EVT-Bench distracted tracking it reaches 85.5% with a single camera (+10.1). With the same encoders, the self-constructed space clearly outperforms concatenating their outputs and learns to weight their layers unevenly, and a dream in it cuts collisions by a quarter. A single dreamed latent decodes into the depth, semantics, and 3D occupancy of the scene a moment ahead, more accurately than pretrained encoders applied to the real future frame, although occupancy is never supervised, and the rollout tracks the robot's own motion over the next second: the dream emerges as a compact latent world model.

Citation

@article{Chen2026_261031762,
  title     = {See What I Need, Dream What I Want: A Navigation Foundation Model on a Self-Constructed Representation Space},
  author    = {Jiawei Chen and Yinxi Wang and Zihan Yang and Meijun Gao and Yaowei Liu and Shuming Zhang and Zecheng Li and Xiangju Mi and Yuxuan Liu and Runzhong Huang and Tianlin Li and Xun Zhang and Zhengyang Wang and Xu Miao and Tong Yang},
  year      = {2026},
  eprint    = {2610.31762},
  archivePrefix = {arXiv},
  primaryClass  = {cs.RO},
  url       = {https://arxiv.org/abs/2610.31762}
}