CARLAverse: A Highly Modular, Distributed, and Multimodal Framework for Human-in-the-Loop Simulation

Summary

Abstract Summary

The development of autonomous driving demands comprehensive testing in mixed-traffic scenarios involving vulnerable road users (VRUs), where purely artificial agents often fail to capture authentic human social negotiations. While human-in-the-loop (HITL) simulators enable safe investigation of these interactions, existing multi-agent platforms struggle with the network latency and synchronization constraints required for high-fidelity haptic feedback. To resolve this, we present CARLAverse, an open-source, multimodal simulation ecosystem. Extending modular hardware abstraction, CARLAverse integrates driving (DrivoCARLA), cycling (CycloCARLA), and pedestrian (WalkoCARLA) simulators into a shared virtual environment. Its core methodological contribution is a distributed physics architecture: latency-critical ego dynamics and high-frequency force feedback are computed locally on client nodes, while a central CARLA server orchestrates non-player character (NPC) physics and global traffic. By decoupling haptic control loops from network bottlenecks, CARLAverse enables scalable, cross-institutional HITL experiments without compromising physical immersion. Code and documentation: https://git.ieem-ka.de/simulator-environments/carlaverse

Key Points

The development of autonomous driving demands comprehensive testing in mixed-traffic scenarios involving vulnerable road users (VRUs), where purely artificial agents often fail to capture authentic human social negotiations.
While human-in-the-loop (HITL) simulators enable safe investigation of these interactions, existing multi-agent platforms struggle with the network latency and synchronization constraints required for high-fidelity haptic feedback.
To resolve this, we present CARLAverse, an open-source, multimodal simulation ecosystem
Extending modular hardware abstraction, CARLAverse integrates driving (DrivoCARLA), cycling (CycloCARLA), and pedestrian (WalkoCARLA) simulators into a shared virtual environment.
Its core methodological contribution is a distributed physics architecture: latency-critical ego dynamics and high-frequency force feedback are computed locally on client nodes, while a central CARLA server orchestrates non-player character (NPC) physics and global traffic.

CARLAverse Architecture

To facilitate a high-fidelity, multi-agent mixed traffic environment without compromising the latency required for HITL immersion, the CARLAverse relies on a specialized, distributed software and hardware architecture. The CARLAverse ecosystem’s core topology follows CARLA ’s strict server-client paradigm, but is extended to support multiple, heterogeneous simulator clients simultaneously.

At the center of the architecture is a dedicated central server running the CARLA Unreal Engine environment.

CARLAverse Architecture — CARLAverse: A Highly Modular, Distributed, and Multimodal Framework for Human-in-the-Loop Simulation — Fig. 1: The CARLAverse multi-server topology. A central global orchestrator synchronizes heterogeneous simulator nodes (e.g., desktop, motion, bicycle, and pedestrian setups) that can be geographically distributed across multiple cooperating research institutions.

Cross-Cutting Frameworks and Orchestration

Creating a cohesive, shared virtual environment for multiple human actors requires more than just distributed physics. To ensure high modularity and valid experimental results across all nodes, the CARLAverse relies on cross-cutting technologies that manage system orchestration, telemetry visualization, and sensory augmentation uniformly across the framework.

Multimodal Simulation Nodes in the CARLAverse

The theoretical architecture of distributed physics and direct API communication forms the backbone of the CARLAverse. However, the physical manifestation of this ecosystem consists of specialized, multimodal simulator nodes. Each node acts as a highly immersive client designed to replicate the specific perceptual cues (visual, auditory, haptic, and vestibular) required for a particular type of road user.

The ecosystem currently encompasses three primary domains: DrivoCARLA (vehicles), CycloCARLA (bicycles), and WalkoCARLA (pedestrians).

Multimodal Simulation Nodes in the CARLAverse — CARLAverse: A Highly Modular, Distributed, and Multimodal Framework for Human-in-the-Loop Simulation — Fig. 4: UI of the CARLAverse launcher. The dashboard features controls for managing the simulator lifecycle, an integrated tree-view editor for intuitive adjustment of module configurations (e.g., control, FFB , motion), and a terminal log panel for real-time system monitoring.

Sources and Demonstrations

Figures are reproduced from Rebling et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.