DeViGrasp: Robust Visual Mobile Grasping for Quadruped Manipulators under Degraded Perception

Summary

Quadruped manipulators enable mobile grasping in complex environments, yet their whole-body control policies remain vulnerable to unreliable onboard visual perception.

Abstract Summary

Quadruped manipulators enable mobile grasping in complex environments, yet their whole-body control policies remain vulnerable to unreliable onboard visual perception. Existing methods are typically developed under relatively reliable observations and have not systematically examined how occlusion, segmentation-mask dropout, depth noise, and target-localization jitter affect grasp reasoning and target tracking. To address this gap, we introduce DeViGrasp-Bench, a benchmark for mobile grasping under degraded vision that incorporates controlled visual degradations, seen and unseen objects, multiple difficulty levels, and complex terrains, and evaluates task success, execution efficiency, and action smoothness. We further propose DeViGrasp-Net, a teacher--student framework that combines state-conditioned grasp reasoning with reliability-aware temporal target estimation. The privileged teacher attends to offline grasp candidates conditioned on object, robot, end-effector, and task states, while the deployable student fuses dual-view segmented-depth observations with current, memory, and recovery target hypotheses through Target Hold Memory and Temporal Memory Attention. DeViGrasp-Net outperforms VBC across degradation levels, unseen objects, and complex terrains, and surpasses an adapted DQ-Net across all evaluated degradation levels. Under the Difficult setting, it achieves a success rate of 62.3\%, improving upon VBC and DQ-Net by 16.1 and 4.3 percentage points, respectively; under the Hard setting, its margin over DQ-Net increases to 10.5 percentage points. Ablation studies confirm the complementary benefits of grasp-aware supervision and reliability-aware temporal memory.

Key Points

Quadruped manipulators enable mobile grasping in complex environments, yet their whole-body control policies remain vulnerable to unreliable onboard visual perception.
Existing methods are typically developed under relatively reliable observations and have not systematically examined how occlusion, segmentation-mask dropout, depth noise, and target-localization jitter affect grasp reasoning and target tracking.
To address this gap, we introduce DeViGrasp-Bench, a benchmark for mobile grasping under degraded vision that incorporates controlled visual degradations, seen and unseen objects, multiple difficulty levels, and complex terrains, and evaluates task success, execution efficiency, and action smoothness.
We further propose DeViGrasp-Net, a teacher--student framework that combines state-conditioned grasp reasoning with reliability-aware temporal target estimation.
The privileged teacher attends to offline grasp candidates conditioned on object, robot, end-effector, and task states, while the deployable student fuses dual-view segmented-depth observations with current, memory, and recovery target hypotheses through Target Hold Memory and Temporal Memory Attention.

DeViGrasp-Bench: Degraded Vision Grasping Benchmark

Quadruped manipulators rely on moving onboard cameras to observe target objects, making visual perception vulnerable to environmental occlusion, arm self-occlusion, segmentation-mask dropout, depth noise, and target-localization errors. To systematically study how these failures affect closed-loop mobile grasping, we introduce DeViGrasp-Bench , a degraded-vision grasping benchmark for quadruped manipulators.

An overview of the benchmark is shown in Fig. 1 . DeViGrasp-Bench is built in Isaac Gym.

DeViGrasp-Bench: Degraded Vision Grasping Benchmark — DeViGrasp: Robust Visual Mobile Grasping for Quadruped Manipulators under Degraded Perception — Figure 2: Overview of DeViGrasp-Net . A privileged teacher learns grasp-aware whole-body actions from object, robot, task, and grasp-candidate information, while a deployable student uses forward/wrist segmented depth observations and temporal memory attention for robust target estimation under degraded perception.

Method

We propose DeViGrasp-Net , a teacher–student framework for robust quadruped mobile grasping under degraded visual perception. As illustrated in Fig. 2 , the framework addresses two complementary challenges: selecting grasp candidates that match the current whole-body state and maintaining a reliable target representation under intermittent visual failures.

Experiments

We evaluate DeViGrasp-Net on DeViGrasp-Bench . The experiments are designed to answer four questions: (1) whether DeViGrasp-Net improves robustness under different levels of visual degradation; (2) how each proposed component contributes to performance; (3) whether the learned policy generalizes to unseen objects; and (4) whether the method remains effective on complex terrain.

All experiments use Isaac Gym on a single NVIDIA RTX 3090 Ti with a Unitree B1–Z1 platform.

Experiments — DeViGrasp: Robust Visual Mobile Grasping for Quadruped Manipulators under Degraded Perception — Figure 4: Qualitative comparison on complex terrains. DeViGrasp-Net achieves stable grasps across rough flat, slope, and stairs, while VBC often fails from target-tracking and end-effector misalignment.

Sources and Demonstrations

Figures are reproduced from Zhou et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.