ENCP: Episode-Normalized Conformal Prediction for Vision-and-Language Navigation

Summary

Abstract Summary

Uncertainty estimation for Vision-Language-Navigation (VLN) models is a critical task since it can help identify ambiguous and unreliable predictions, enabling agents to make safer navigation decisions. As one of the most advanced uncertainty estimation frameworks, conformal prediction (CP) offers a promising approach for uncertainty estimation in VLN. However, given that VLN agent requires a sequence of steps, standard calibration in conformal prediction fails to provide coverage guarantee it promises over a dependent, variable-length VLN episode. To this end, we propose Episode-Normalized Conformal Prediction (ENCP), which rescales a nonconformity score by the policy's residual confidence and calibrates one maximum score per episode. Under exchangeable calibration and test episodes, this construction covers the ground truth at every step with probability at least $1 - α$, while allowing dependence among steps within an episode. Across four VLN policies and three nonconformity scores on R2R and REVERIE dataset, ENCP meets all reported empirical step-coverage targets on the seen-to-unseen evaluation. These results demonstrate that ENCP can provide model-agnostic uncertainty estimates, which might be useful for determining when a VLN agent should defer to a more capable predictor, including human assistance.

Key Points

Uncertainty estimation for Vision-Language-Navigation (VLN) models is a critical task since it can help identify ambiguous and unreliable predictions, enabling agents to make safer navigation decisions.
As one of the most advanced uncertainty estimation frameworks, conformal prediction (CP) offers a promising approach for uncertainty estimation in VLN.
However, given that VLN agent requires a sequence of steps, standard calibration in conformal prediction fails to provide coverage guarantee it promises over a dependent, variable-length VLN episode.
To this end, we propose Episode-Normalized Conformal Prediction (ENCP), which rescales a nonconformity score by the policy's residual confidence and calibrates one maximum score per episode.
Under exchangeable calibration and test episodes, this construction covers the ground truth at every step with probability at least $1 - α$, while allowing dependence among steps within an episode.

Problem Formulation

In this section, we formulate VLN as sequential classification in discrete graph, define the 3 nonconformity scores used in commonly used CP, and show why calibration over pooled steps does not yield trajectory-level coverage. An experimental condition is one policy–dataset pair. An episode is one attempt to follow an instruction $L$ from a given start node.

The agent moves on an undirected graph of panoramic Matterport3D viewpoints [ 21 ] and terminates by selecting stop .

Episode-Normalized Conformal Prediction

Standard CP calibration fails to provide coverage guarantees in sequential VLN due to intra-episode step dependencies and dynamic action spaces.

To resolve this, Episode-Normalized Conformal Prediction (ENCP) derives its name from two core mechanisms: episode-level calibration , which collapses dependent trajectory steps into a single worst-case metric to restore finite-sample guarantees, and score normalization , which adjusts nonconformity scores relative to step-varying policy confidence.

Episode-Normalized Conformal Prediction — ENCP: Episode-Normalized Conformal Prediction for Vision-and-Language Navigation — Fig. 3: ENCP calibrates one worst-step score per episode, then asks for help when the deployed set exceeds $\tau$.

Coverage Guarantee

This section proves coverage for parameter-free ENCP. For this purpose, before calibration, we freeze the navigation policy, a base score $s_{\mathrm{base}}$, and $w_{\mathrm{pf}}(x_{t})=1-p_{\max}(x_{t})$. For episode $E_{i}$, let $x_{i,t}$ and $a_{i,t}^{*}$ denote the decision context and ground-truth action at step $t$, and let $T_{i}=T(E_{i})\geq 1$.

Throughout this section, $s_{\mathrm{norm}}$ denotes the parameter-free score.

Coverage Guarantee — ENCP: Episode-Normalized Conformal Prediction for Vision-and-Language Navigation — Fig. 4: Parameter-free ENCP attains nominal trajectory coverage under episode exchangeability but not under seen-to-unseen shift. Error bars are empirical 95% ranges over 300 exchangeable splits; the shifted bars lie outside Theorem 1 ’s premise.

Experiments

We calibrate on the R2R validation-seen, whose buildings occur in training, and test on validation-unseen (val-unseen) episodes from disjoint building scans [ 1 ] . We also experiment this in REVERIE [ 23 ] for a better generalization. We evaluate four VLNs from three architecture families: DUET [ 9 ] , HAMT [ 8 ] , and Recurrent VLN-BERT with PREVALENT and OSCAR initialization (R-prev and R-osc) [ 7 ] .

Our runs reproduce each published val-unseen success rate to within one percentage point.

Experiments — ENCP: Episode-Normalized Conformal Prediction for Vision-and-Language Navigation — Fig. 5: Simulation of ENCP help-seeking: when the prediction set exceeds the size budget, the simulator supplies the ground-truth action instead of the policy-selected action.

Sources and Demonstrations

Figures are reproduced from Feliren et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.