CrossDepth: Geometry-Constrained Attention for Generalizable Multi-View Surround Depth Estimation

Summary

Abstract Summary

Reliable 3D understanding of the surrounding environment is a core requirement for autonomous driving. Multi-view surround camera rigs provide broad scene coverage, but the spatially adjacent images typically overlap only minimally. Consequently, the depth of most pixels must be inferred from monocular appearance cues. These cues can appear differently across images and may therefore be interpreted differently by the depth estimation model. We target two main sources of cross-image inconsistency: differences in camera intrinsics and the limited receptive field of each image. We address the former by conditioning the features on per-pixel camera-aware ray embeddings, enabling the network to account for camera-dependent variations in monocular cues. We address the latter by extending each pixel's context beyond its own image through cross-image attention constrained to geometrically plausible regions, derived from the calibrated rig setup. The model is trained in a fully self-supervised manner based on photometric consistency. Evaluations on DDAD and nuScenes show improved overall depth accuracy and cross-image depth consistency over state-of-the-art self-supervised methods under in-domain and cross-domain evaluation. Code is available at https://abualhanud.github.io/CrossDepthPage/.

Key Points

Reliable 3D understanding of the surrounding environment is a core requirement for autonomous driving.
Multi-view surround camera rigs provide broad scene coverage, but the spatially adjacent images typically overlap only minimally.
Consequently, the depth of most pixels must be inferred from monocular appearance cues
These cues can appear differently across images and may therefore be interpreted differently by the depth estimation model.
We target two main sources of cross-image inconsistency: differences in camera intrinsics and the limited receptive field of each image.

Methodology

Given a surround-view camera rig consisting of $N$ time-synchronized cameras with overlapping fields of view, known intrinsic calibration parameters, and metrically scaled relative poses, the objective is to estimate a metric depth map for each input image. Our depth estimation network follows an encoder-decoder architecture, inspired by [ 31 , 51 ] .

For frame $t$, the input images are denoted by $\mathbf{I}_{t}\in\mathbb{R}^{N\times H\times W\times 3}$, where $H$ and $W$ denote the image height and width, respectively.

Methodology — CrossDepth: Geometry-Constrained Attention for Generalizable Multi-View Surround Depth Estimation — Figure 3 : Visualization of the flattened cylindrical representation, where overlapping image regions are mapped to nearby locations. For a query token (red star), candidate tokens in the other image are retrieved from a local neighborhood window on the cylinder (blue box) that spans multiple images. Valid cross-image candidates within this window are shown as blue dots in the other overlapping image.

Experiments

We compare our method with five state-of-the-art self-supervised approaches: FSM [ 16 ] , SurroundDepth [ 47 ] , VFDepth [ 22 ] , CVCDepth [ 6 ] , and CylinderDepth [ 1 ] . We report results for the entire image and the overlapping regions under in-domain and cross-domain evaluation.

We further conduct ablation studies to assess the effects of cross-image attention, geometric attention constraints, and camera-aware ray embeddings. Experiments are conducted on two surround-view datasets, DDAD [ 15 ] and nuScenes [ 4 ] .

Experiments — CrossDepth: Geometry-Constrained Attention for Generalizable Multi-View Surround Depth Estimation — Figure 4 : Comparison of depth maps predicted by our method and by state-of-the-art methods on DDAD. Our results show better preserved details and well-defined object boundaries (green bounding boxes). Depth is shown from close in yellow to distant in blue.

Sources and Demonstrations

Figures are reproduced from Abualhanud et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.