Generalizable Robustness Testing of DNN-Based Robotic Navigation Systems via XAI-Guided Search

Summary

**Context:** Deep Neural Networks (DNNs) increasingly control Cyber-Physical Systems (CPSs), yet small input perturbations can cause unsafe system-level behavior.

Abstract Summary

**Context:** Deep Neural Networks (DNNs) increasingly control Cyber-Physical Systems (CPSs), yet small input perturbations can cause unsafe system-level behavior. Existing approaches often optimize perturbations for individual images and evaluate them only in simulation, limiting their generalizability and practical validity. **Objectives:** This work aims to generate robustness tests that remain effective across operational observations and to evaluate whether the resulting failures transfer from simulation to a physical robot. **Methods:** We propose an explainability-guided multi-objective evolutionary approach that generates sparse perturbations over representative images selected through visual and behavioral clustering. Aggregated Integrated Gradients guide mutations toward influential image regions. We evaluate the approach on a DNN-controlled LeoRover in Gazebo, conduct an ablation study, and validate a stratified subset of perturbations on the physical robot. **Results:** The approach achieved a median success rate of 70.0%, compared with 53.85% for unguided search, and increased median hypervolume from 0.65 to 0.73. Multi-image optimization improved the success rate from 50.0% to 57.5%, while XAI guidance further increased it to 70.0%. In the sim-to-real evaluation, simulation achieved 0.95 precision and 0.67 recall, and simulated and physical failure times showed a significant positive correlation of 0.617. **Conclusion:** Combining multi-image optimization with explainability-guided search improves robustness testing for DNN-controlled robotic systems. Simulation effectively identifies and prioritizes transferable failures, but physical validation remains necessary because some real-world failures are not reproduced in simulation.

Key Points

**Context:** Deep Neural Networks (DNNs) increasingly control Cyber-Physical Systems (CPSs), yet small input perturbations can cause unsafe system-level behavior.
Existing approaches often optimize perturbations for individual images and evaluate them only in simulation, limiting their generalizability and practical validity.
**Objectives:** This work aims to generate robustness tests that remain effective across operational observations and to evaluate whether the resulting failures transfer from simulation to a physical robot.
**Methods:** We propose an explainability-guided multi-objective evolutionary approach that generates sparse perturbations over representative images selected through visual and behavioral clustering.
Aggregated Integrated Gradients guide mutations toward influential image regions

XAI-guided search-based robustness testing

Our approach consists of three sequential phases. First, instead of generating perturbations from a single input image (as conducted in the literature [ 33 , 27 ] ), we extract a compact set of representative images from the operational dataset, since single-image perturbation-based tests may overfit to one specific scene and may not reflect the diversity of situations encountered by the robot.

Experimental Setup

This section explains the experimental setup of our approach when applying it into an autonomous robotic platform. Specifically, we aim to answer the following three research questions (RQs): RQ1 – Sanity Check: Does XAI-guided evolutionary search outperform unguided search in generating high-impact perturbations? This RQ serves as a sanity check for the overall search strategy.

Threats to Validity

We now discuss the threats of our study and the actions we took to try to mitigate them. External validity: Our evaluation considers one DNN controller, one robotic platform, and one navigation task, which limits the generalization of the results to other models, robots, and operational environments. We partially mitigate this threat by evaluating the perturbations in both simulation and the physical system.

Internal validity: The selected clustering, attribution, perturbation, and search parameters may influence the results.

Threats to Validity — Generalizable Robustness Testing of DNN-Based Robotic Navigation Systems via XAI-Guided Search — (a) angular velocity = 0.61

Sources and Demonstrations

Figures are reproduced from Sohail et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.