PIVOT: Physically Informed Vision-Language Off-Road Traversability for Field Robot Navigation

Summary

Abstract Summary

Terrain assessment is a critical capability for off-road mobile robots, enabling safe and reliable navigation through unstructured and geometrically complex environments. Conventional geometry-based terrain assessment is fast to compute but often overly conservative in unstructured environments. We present PIVOT: a Physically Informed Vision-Language Off-Road Traversability navigation system that augments conventional geometry-based planning with vision-language-model (VLM)-based semantic reasoning for field robots. To physically ground this assessment, we quantify how strongly the VLM's predicted traversal energy cost, robot vibration, and wheel slip correlate with real-world measurements and introduce a unified traversability score that weights each modality by its prediction-measurement correlation. For efficiency, we design a two-level navigation architecture that retains geometry-based planning as the nominal mode and invokes semantic replanning only when that mode fails to find a path. Across five repeated closed-loop trials on a mixed-terrain route totalling around $6.4$ km, the proposed system increases overall autonomy from $59.6\%$ to $97.0\%$, reduces human interventions from $11$ to $3$, and increases the mean distance between interventions (MDBI) from $69.2$ m to $412.9$ m compared with geometry-only navigation. These results demonstrate that physically grounded VLM-based terrain assessment can substantially extend autonomous navigation beyond the limitations of geometry alone, while preserving efficient geometric planning as the nominal mode.

Key Points

Terrain assessment is a critical capability for off-road mobile robots, enabling safe and reliable navigation through unstructured and geometrically complex environments.
Conventional geometry-based terrain assessment is fast to compute but often overly conservative in unstructured environments.
We present PIVOT: a Physically Informed Vision-Language Off-Road Traversability navigation system that augments conventional geometry-based planning with vision-language-model (VLM)-based semantic reasoning for field robots.
To physically ground this assessment, we quantify how strongly the VLM's predicted traversal energy cost, robot vibration, and wheel slip correlate with real-world measurements and introduce a unified traversability score that weights each modality by its prediction-measurement correlation.
For efficiency, we design a two-level navigation architecture that retains geometry-based planning as the nominal mode and invokes semantic replanning only when that mode fails to find a path.

Methodology

The proposed system diagram is shown in Figure 2 . We adopt a two-level navigation architecture in which geometry-based planning serves as the nominal mode and VLM-based semantic reasoning provides a fallback when geometric planning cannot identify a feasible path. Both levels operate on a local costmap constructed from the LiDAR observations.

The nominal geometry-based terrain assessment constructs a local traversability costmap directly from the LiDAR point cloud.

Methodology — PIVOT: Physically Informed Vision-Language Off-Road Traversability for Field Robot Navigation — Fig. 2: The proposed two-level navigation architecture, consisting of nominal geometry-based planning and semantic fallback planning, with human intervention serving as an external recovery mechanism. The C-BIT ∗ planner first plans on the geometry-based costmap. If no valid plan is found, the system performs VLM-based terrain assessment and replans on a semantic costmap; persistent failure leads to human intervention. Dashed transitions indicate a return to nominal planning after recovery or when a new waypoint is provided.

Experiments

In this section, we present an offline evaluation of the proposed physically grounded terrain assessment and a comprehensive online closed-loop evaluation of PIVOT.

Using an extensive offline dataset, we first establish positive correlations between the three VLM predictions and their corresponding physical measurements, as well as a positive correlation between the proposed traversability score and a reference score obtained by averaging the three normalized measurements.

Experiments — PIVOT: Physically Informed Vision-Language Off-Road Traversability for Field Robot Navigation — Fig. 3: Sample LiDAR inputs used for semantic terrain assessment. The cyan highlighting denotes the region of interest for terrain assessment and tells the VLM where the robot is expected to traverse next.

Sources and Demonstrations

Figures are reproduced from Jiao et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.