FoundationVIO: Enforcing Metric Scale in Feed-forward Visual-Inertial Odometry

Summary

Abstract Summary

Feed-forward geometric foundation models (GFMs) directly predict dense geometry and camera poses from batches of images. Emerging visual SLAM methods chain together consecutive GFM batches to form dense point cloud maps. A core challenge is how to address the ambiguous geometric scale of GFM predictions: independent predictions are locally consistent up-to-scale but not constrained by any direct physical measurements. Models trained on data with known scale improve the consistency of GFM reconstructions but again do not directly measure it. Consequently, trajectories and 3D reconstructions from GFM-enabled SLAM methods inherit this issue. Motivated by parallels with classical visual-inertial odometry, we introduce FoundationVIO which integrates inertial sensing with GFM predictions to recover metric scale reconstruction using a factor-graph based approach. We demonstrate odometry estimates competitive with classical state-of-the-art VIO methods and accurate dense reconstructions that outperform GFM-enabled SLAM methods. We also add a minimal GFM-based loop-closure mechanism to create a full Visual-Inertial SLAM system capable of creating globally accurate reconstructions and pose estimates. We evaluate our method on three datasets with well known sensor systems and demonstrate its compatibility on ubiquitous camera and IMU sensors. The method runs at approximately real-time on a consumer-grade laptop GPU. Code: https://github.com/ori-drs/FoundationVIO.

Key Points

Feed-forward geometric foundation models (GFMs) directly predict dense geometry and camera poses from batches of images.
Emerging visual SLAM methods chain together consecutive GFM batches to form dense point cloud maps
A core challenge is how to address the ambiguous geometric scale of GFM predictions: independent predictions are locally consistent up-to-scale but not constrained by any direct physical measurements.
Models trained on data with known scale improve the consistency of GFM reconstructions but again do not directly measure it.
Consequently, trajectories and 3D reconstructions from GFM-enabled SLAM methods inherit this issue

Method

An overview of the proposed method is shown in Fig. 2 . We propose a factor-graph based design that takes IMU measurements, up-to-scale poses from GFM predictions, and relative scale measurements derived between consecutive predictions in order to create a natively metric scale trajectory and 3D reconstruction.

Evaluation

We evaluate both the trajectory and reconstruction quality of FoundationVIO across three datasets. We first compare FoundationVIO’s pose estimation accuracy with respect to both classical and GFM-enabled methods. Results are presented both with the system configured in odometry mode and also in SLAM mode with loop-closures enabled.

We then evaluate FoundationVIO’s dense 3D reconstructions against other state-of-the-art GFM-enabled SLAM methods. Finally, we evaluate the real-time performance of our method.

Evaluation — FoundationVIO: Enforcing Metric Scale in Feed-forward Visual-Inertial Odometry — Fig. 2: Component diagram of the proposed system. The factor graph contains per-batch scale $l_{k}=\log{s_{k}}$ and per-frame pose $\mathbf{T}_{i}$, velocity $\mathbf{v}_{i}$, and biases $\mathbf{b}_{i}$ as variables. IMU data, up-to-scale GFM pose predictions, and the batch-to-batch relative scale estimates are used as measurements for the IMU preintegration factor (red), GFM pose factor (yellow), and scale consistency factor (blue).

Sources and Demonstrations

Figures are reproduced from Rhee et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.