Compact Latent Coordination for Autonomous Vehicles at Unsignalized Intersections

Summary

Abstract Summary

Coordinating autonomous vehicles at unsignalized intersections remains a critical challenge for multi-agent reinforcement learning (MARL) systems, which typically struggle with combinatorial action spaces, reliance on privileged information, or rigid agent designs. We propose Master-Agent Proto-plan System (MAPS), a hierarchical deep reinforcement learning (DRL) architecture in which a centralized Master agent generates a compact, continuous embedding, denoted as proto-plan, that encodes a global coordination strategy. Decentralized Worker agents integrate this embedding with local observations to execute vehicle-specific control, decoupling strategic intent from tactical execution and enabling independent optimization of each module. As a proof-of-concept evaluation of this coordination mechanism, we test MAPS across 72 intersection configurations in HighwayEnv. MAPS achieves collision-free navigation while significantly reducing average travel time, outperforming state-of-the-art baselines. The learned proto-plans further exhibit robust generalization: a system trained with three agents achieves a 94% success rate when deployed zero-shot to five-agent scenarios, confirming that proto-plan-based hierarchical learning provides a promising framework for multi-vehicle coordination.

Key Points

Coordinating autonomous vehicles at unsignalized intersections remains a critical challenge for multi-agent reinforcement learning (MARL) systems, which typically struggle with combinatorial action spaces, reliance on privileged information, or rigid agent designs.
We propose Master-Agent Proto-plan System (MAPS), a hierarchical deep reinforcement learning (DRL) architecture in which a centralized Master agent generates a compact, continuous embedding, denoted as proto-plan, that encodes a global coordination strategy.
Decentralized Worker agents integrate this embedding with local observations to execute vehicle-specific control, decoupling strategic intent from tactical execution and enabling independent optimization of each module.
As a proof-of-concept evaluation of this coordination mechanism, we test MAPS across 72 intersection configurations in HighwayEnv.
MAPS achieves collision-free navigation while significantly reducing average travel time, outperforming state-of-the-art baselines.

Proposed Approach

We formulate multi-vehicle intersection coordination as a hierarchical Markov decision process comprising two levels (Figure 1 ): a centralized Master agent that observes the global traffic state and generates a proto-plan embedding $z_{t}\in\mathbb{R}^{d}$ representing the desired coordination strategy, and decentralized Worker agents , each controlling an individual vehicle, that combine the proto-plan with local observations to select vehicle-specific actions.

Experimental Setup

We evaluate our approach using the HighwayEnv simulator [ Leurent(2018) ] , an open-source reinforcement learning environment for autonomous driving research. Its modular architecture permits customization of state representations, action spaces, and reward functions, making it well-suited to our hierarchical framework.

Evaluation Results

Table 1 summarizes the comparative results, and Figure 3 illustrates the learning dynamics during training. MAPS is the only approach to achieve collision-free evaluation while recording the lowest average travel time. MAPS achieves zero collisions during evaluation (100% success rate), compared to 35 collisions (65% SR) for VN-MADDPG and 31 collisions (69% SR) for QMIXwD. The safety advantage extends to training.

Sources and Demonstrations

Figures are reproduced from Lifshits et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.