Robotics Research Hub
Context-Aware RGB-Thermal-Radar Obstacle Perception for Resource-Constrained ASVs

Context-Aware RGB-Thermal-Radar Obstacle Perception for Resource-Constrained ASVs

Pranav Kedia, Edwin Redhead, Heiko Hamann · N/A · 2026

vision reinforcement-learning planning control
PDF arXiv

Framework

N/A

License

N/A

Stars

N/A

Summary

Reliable obstacle perception is challenging for small autonomous surface vessels (ASVs): reflections and glare confound visible-light cameras, vessel motion shifts the apparent waterline, and darkness removes the dominant sensing modality altogether.

Abstract Summary

Reliable obstacle perception is challenging for small autonomous surface vessels (ASVs): reflections and glare confound visible-light cameras, vessel motion shifts the apparent waterline, and darkness removes the dominant sensing modality altogether. Rather than treating multimodal fusion as a static accuracy gain, we propose a context-aware architecture that combines RGB and long-wave infrared (LWIR) imagery, frequency-modulated continuous-wave (FMCW) mmWave radar, and inertial context. A bearing-sector interface provides the navigation controller with calibrated obstacle probabilities, radar ranges, and sensor status. It also distinguishes sensor dropout from a valid observation with no detected obstacle. Using new multimodal lake recordings, we deployed the pipeline in a 24-leg closed-loop pilot under uncontrolled lake conditions (16 day, 4 dusk, and 4 night; 351 m), including night legs in which RGB was unusable and the vessel drove on LWIR and radar evidence alone. On eight held-out low-light recordings, the calibrated fusion model achieved average precision (AP) 0.869 and expected calibration error 0.108, compared with AP 0.673 for rule-based fusion, 0.620 for gated fusion, and 0.903 for RGB alone. Our results expose a central tradeoff: RGB ranks best when usable, whereas fusion preserves a navigation input when RGB fails. This work is a step toward perception that remains usable and degrades predictably across multi-day ASV missions.

Key Points

  • Reliable obstacle perception is challenging for small autonomous surface vessels (ASVs): reflections and glare confound visible-light cameras, vessel motion shifts the apparent waterline, and darkness removes the dominant sensing modality altogether.
  • Rather than treating multimodal fusion as a static accuracy gain, we propose a context-aware architecture that combines RGB and long-wave infrared (LWIR) imagery, frequency-modulated continuous-wave (FMCW) mmWave radar, and inertial context.
  • A bearing-sector interface provides the navigation controller with calibrated obstacle probabilities, radar ranges, and sensor status.
  • It also distinguishes sensor dropout from a valid observation with no detected obstacle
  • Using new multimodal lake recordings, we deployed the pipeline in a 24-leg closed-loop pilot under uncontrolled lake conditions (16 day, 4 dusk, and 4 night; 351 m), including night legs in which RGB was unusable and the vessel drove on LWIR and radar evidence alone.

System and Operational Overview

Our perception module combines RGB, LWIR thermal, and FMCW mmWave radar measurements with inertial context on an embedded computer. Its output provides the navigation controller with bearing-resolved representation of detected obstacles. Next, we describe the method’s operating constraints, sensor configuration, and timing/geometric references.

The perception subsystem consumes energy through sensing, computation, interfaces, and power conversion.

System and Operational Overview — Context-Aware RGB-Thermal-Radar Obstacle Perception for Resource-Constrained ASVs
Fig. 1: Aerial view of the field trial showing the ASV, kayak, and two people.

Multimodal Perception and Sector Fusion

We first undistort each RGB frame and use an embedded segmentation network to classify pixels as obstacle, water, or sky. We trained an LR-ASPP/MobileNetV3 student network through knowledge distillation [ 14 ] from eWaSR [ 4 ] , using LaRS [ 1 ] and 10,731 frames from our own recordings labeled by the teacher.

Multimodal Perception and Sector Fusion — Context-Aware RGB-Thermal-Radar Obstacle Perception for Resource-Constrained ASVs
Fig. 2: Overview of our multimodal perception module. Synchronized RGB, thermal, radar, and inertial measurements produce modality-specific evidence. Context-conditioned fusion converts this evidence into a compact bearing-sector representation for the navigation controller.

Data, Labels and Experimental Design

Recordings were made on a lake (Fig. 1 ) from our ASV, a pontoon, and a moored vessel, between June and September 2026: six daylight, two twilight, and one night session ($13.0\text{\,}\mathrm{h}$, 121,313 synchronized frames in the curated corpus), with buoys, moored and moving boats, wading and swimming people, water birds, and shoreline structures at ranges from $1\text{\,}\mathrm{m}$ to the shore.

Illumination is measured: exposure time, gain, and luminance are logged per frame, and sun elevation is computed from RTC time and GPS position.

Data, Labels and Experimental Design — Context-Aware RGB-Thermal-Radar Obstacle Perception for Resource-Constrained ASVs
Fig. 3: Left: the perception box. Right: monocular range error on an isolated wading target at 6.9–$8.9\text{\,}\mathrm{m}$ before and after the afloat calibration (camera height 0.27$\rightarrow$$0.25\text{\,}\mathrm{m}$, IMU attitude, water-line): median absolute error 8.3% $\rightarrow$ 3.5%, 73% $\rightarrow$ 100% of detections within the $\pm$20% gate ($n=11$).

Results

After the afloat calibration (camera height re-measured under load, IMU attitude, water-line contact point), monocular range on an isolated wading target at 6.9 to $8.9\text{\,}\mathrm{m}$ has a median absolute error of $3.5\text{\,}\mathrm{\%}$ (mean $+1.3\%$, 100% within $\pm 20\%$, $n=11$), against 8.3%/73% at the out-of-water height (Fig. 3 ). The height correction accounted for most of the gain.

Over whole dockside clips, the error stays large because the camera sees the shore beyond the radar’s $9\text{\,}\mathrm{m}$ range.

Results — Context-Aware RGB-Thermal-Radar Obstacle Perception for Resource-Constrained ASVs
Fig. 4: Day-to-night recording from 17:11 to 21:36. Top: solar elevation and RGB luminance. Bottom: per-sensor evidence fractions (60 s medians) and the fraction of onboard updates meeting the mask-age criterion. Gray marks RGB rejection; blue marks night trials.

Sources and Demonstrations

Figures are reproduced from Kedia et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.

Share

Related Papers

Aim Short to Reach Far: Your Frozen World Model Can Plan Better Than You Think
arXiv preprint

Aim Short to Reach Far: Your Frozen World Model Can Plan Better Than You Think

Xvyuan Liu, Jianjie Fang, Chen Gao et al.

Planners built on visual world models commonly score each predicted outcome by its distance to the encoded goal image. We show that this target can limit control even with exact dynamics and globally optimal short-horizon search: reaching a goal may require actions that initially move away from it.

vision reinforcement-learning planning control human-robot-interaction benchmark
PDF Intermediate
Autonomous Droplet Navigation via Model-Based Reinforcement Learning: Zero-Shot Transfer and Emergent Dynamics
arXiv preprint

Autonomous Droplet Navigation via Model-Based Reinforcement Learning: Zero-Shot Transfer and Emergent Dynamics

Rajneesh Anand, Mayuresh V. Kothare

Self-driving laboratories (SDLs) are transforming chemical and materials discovery through closed-loop automation, yet automated infrastructure for physical manipulation of soft, deformable matter remains beyond current robotic platforms.

manipulation vision reinforcement-learning planning control
Code PDF Intermediate
GitHub ★ —