LANTERN: A Closed-Loop Benchmark for VLM-Based Cooperative Driving with Temporally Grounded Warnings

Yongshuo Liu, Xu Gao, Morui Zhu, Yongqi Zhu, Qi Chen, Deyuan Qu, Song Fu, Qing Yang · N/A · 2026

Summary

We present LANTERN, a closed-loop benchmark for temporally grounded cooperative warnings.

Abstract Summary

We present LANTERN, a closed-loop benchmark for temporally grounded cooperative warnings. LANTERN separates warning onset, hazard onset, warning termination, and post-hazard recovery, and evaluates each physical event under matched warning and no-warning executions so that the warning's contribution is measured in isolation rather than confounded with onboard vision. The benchmark spans six safety-critical scenario families and provides 3,272 sequences with 236,309 frames for training, together with 120 matched route pairs for closed-loop evaluation. Each hazard route is evaluated under the warning and no-warning conditions, while its no-hazard control penalizes unconditional braking. We further introduce the Cooperative Unified Score (CUS), a safety-gated metric that jointly rewards route progress, anticipation, clearance, and recovery. Fine-tuning a representative VLM driving model raises CUS from 34.6 without warnings to 75.5 with them, demonstrating both the value of cooperative warnings and the discriminative power of the paired protocol. All resources will be made publicly available.

Key Points

We present LANTERN, a closed-loop benchmark for temporally grounded cooperative warnings
LANTERN separates warning onset, hazard onset, warning termination, and post-hazard recovery, and evaluates each physical event under matched warning and no-warning executions so that the warning's contribution is measured in isolation rather than confounded with onboard vision.
The benchmark spans six safety-critical scenario families and provides 3,272 sequences with 236,309 frames for training, together with 120 matched route pairs for closed-loop evaluation.
Each hazard route is evaluated under the warning and no-warning conditions, while its no-hazard control penalizes unconditional braking.
We further introduce the Cooperative Unified Score (CUS), a safety-gated metric that jointly rewards route progress, anticipation, clearance, and recovery.

The LANTERN Benchmark

Driving task and warning interface. LANTERN evaluates an autonomous driving model $f_{\theta}$ in closed loop. At time $t$, the model receives visual observations $I_{t}$, ego state $s_{t}$, navigation state $r_{t}$, and an optional cooperative warning $c_{t}$, and predicts an $H$-step future ego trajectory, where $c_{t}=\varnothing$ when no warning is active.

The benchmark rolls out these predictions in CARLA and measures the resulting safety, route progress, and recovery.

The LANTERN Benchmark — LANTERN: A Closed-Loop Benchmark for VLM-Based Cooperative Driving with Temporally Grounded Warnings — Figure 2: Temporal organization of a LANTERN scenario. The warning begins before physical-event onset and terminates when the hazard is first directly observed through the ego vehicle’s onboard sensors.

Experiments

The primary evaluation uses the 120 fixed Route A/B pairs defined in Section 3.4 , with 20 valid pairs per scenario family and driving model. Our cooperative baseline, Coop-SimLingo , fine-tunes the original SimLingo checkpoint ( Renz et al. 2025 ) on LANTERN without architectural changes; its action-coupled language interface allows cooperative warnings to affect driving directly.

Sources and Demonstrations

Figures are reproduced from Liu et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.