M3GD: Multi-Modal Multi-View Geometric Diffusion for Camera--LiDAR Novel View Synthesis

Yang Zhou, Jiuhong Xiao, Shizhao Ye, Long Quang, Carlos Nieto-Granda, Giuseppe Loianno · N/A · 2026

Summary

Abstract Summary

Robotic novel view synthesis (NVS) must recover both visual appearance and metric 3D structure, yet most generative NVS methods rely only on images, overlooking LiDAR, a complementary sensor common on robotic platforms. We present M3GD, a Camera--LiDAR multimodal representation for generative NVS that composes independently pretrained 2D image and 3D point-cloud foundation models without separately pretraining a cross-modal translator. We show that, after camera projection, frozen LiDAR and image features exhibit substantial shared spatial structure, providing a natural cross-modal representation. M3GD conditions generation on LiDAR through this structure: it combines explicit geometry statistics with learned point-cloud descriptors into view-aligned packets on the image-latent grid, injected through a lightweight residual adapter into a multi-view flow-matching generator whose latent space, decoders, and training objective remain intact. On the GrandTour dataset, M3GD improves target-view RGB and depth synthesis over an image-only version of the same backbone. Ablations show that the gains come from pixel-aligned LiDAR content and that target-view LiDAR acts as a geometric query linking the requested view to source observations. Deployment on a ground robot demonstrates practical real-world operation, with a configurable quality--cost trade-off controlled by the number of Euler integration steps.

Key Points

Robotic novel view synthesis (NVS) must recover both visual appearance and metric 3D structure, yet most generative NVS methods rely only on images, overlooking LiDAR, a complementary sensor common on robotic platforms.
We present M3GD, a Camera--LiDAR multimodal representation for generative NVS that composes independently pretrained 2D image and 3D point-cloud foundation models without separately pretraining a cross-modal translator.
We show that, after camera projection, frozen LiDAR and image features exhibit substantial shared spatial structure, providing a natural cross-modal representation.
M3GD conditions generation on LiDAR through this structure: it combines explicit geometry statistics with learned point-cloud descriptors into view-aligned packets on the image-latent grid, injected through a lightweight residual adapter into a multi-view flow-matching generator whose latent space, decoders, and training objective remain intact.
On the GrandTour dataset, M3GD improves target-view RGB and depth synthesis over an image-only version of the same backbone.

Problem Statement and Preliminaries

To ground the projected Camera–LiDAR conditioning used by M3GD, we first state the NVS problem, subsequently introduce the diffusion and flow-matching machinery used by our backbone, and finally summarize the 2D and 3D foundation representations composed by M3GD.

Methodology

M3GD composes 2D and 3D representations through a projected RGB–LiDAR conditioning pathway (Fig. 1 ). The frozen DA3 encoder maps the RGB image $I$ to an $H{\times}W$ latent grid $z$, the space in which the inherited GLD backbone generates novel views.

In parallel, the frozen Utonia encoder assigns a descriptor to every point of the LiDAR sweep $P$, and the camera projection $\pi$ pools these descriptors into the cells of a second grid $h$ with the same layout.

Methodology — M3GD: Multi-Modal Multi-View Geometric Diffusion for Camera--LiDAR Novel View Synthesis — Fig. 2: Overview of M3GD. (a) Posed source views provide RGB, LiDAR, and sensor poses, while the target query supplies only a camera pose and LiDAR; target RGB is never observed. (b) The frozen DA3 encoder maps source images to image latents. From the same LiDAR sweep, camera projection constructs explicit geometry statistics, while the frozen Utonia encoder supplies learned per-point descriptors that are pooled into the corresponding image-latent cells; their concatenation forms the per-view LiDAR packet. Fig. 1 details this projected RGB–LiDAR representation. A zero-initialized residual adapter composes this packet with the DA3 grid cell by cell. (c) The resulting source tokens (RGB + LiDAR) and target tokens (noise + LiDAR query) enter the multi-view flow matching network, which integrates the learned velocity field for $K$ Euler steps to produce the target latent; frozen decoders recover RGB and depth. Arrow colors distinguish the image-appearance (blue), metric-geometry (teal), and generation (purple) pathways.

Experimental Setup

This section fixes the protocol shared by all experiments in Section VI : implementation (Section V-A ), dataset and metrics (Section V-B ), baselines (Section V-C ), and the real-robot platform (Section V-D ). Backbones. We leverage the released GLD checkpoint [ 8 ] , whose frozen encoder and decoders are described in Section IV-B .

Each DA3 feature level synthesized by the diffusion model is a $1536$-channel map ($C{=}1536$) on the image patch grid.

Experimental Setup — M3GD: Multi-Modal Multi-View Geometric Diffusion for Camera--LiDAR Novel View Synthesis — Fig. 3: Clearpath Warthog platform used for the real-robot deployment.

Results

Under the protocol of Section V , we first test the representation-level premise of M3GD (Section VI-A ). We then present the baseline comparison and mechanism ablations (Sections VI-B and VI-D ), qualitative analysis (Section VI-C ), and the deployment-recipe, real-robot, and runtime results (Sections VI-E – VI-G ).

Sources and Demonstrations

Figures are reproduced from Zhou et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.