LEAP: Making Privileged Geometry Supervision Effective for Visuomotor Learning

Han Fang, Yunpeng Jiang, Jianshu Hu, Zhiyuan Guan, Ruiguo Sun, Shujia Li, Paul Weng, Xiao Li, Yutong Ban · N/A · 2026

Summary

Abstract Summary

Privileged 3D supervision uses additional geometric information during training to guide RGB-based visuomotor policy learning, without requiring geometric inputs at deployment. However, low reconstruction error does not ensure that visual representations capture geometry useful for control. We identify three limitations that weaken this supervision: proprioceptive shortcuts, dominant-view reliance, and reconstruction objectives dominated by task-irrelevant geometry. To address these limitations, we propose Latent Encoding with Aligned Privileged Geometry (LEAP). Our framework uses an auxiliary decoder to reconstruct point clouds within the manipulation workspace from visual features alone, while retaining proprioception for action prediction. Alongside full reconstruction, we introduce wrist-view dropout and partial reconstruction targets matched to the retained wrist, encouraging the encoder to capture complementary local geometry. The auxiliary decoder is removed at inference, leaving only RGB observations and robot state as policy inputs. Extensive experiments on RoboTwin, ManiSkill, and real-world tasks demonstrate consistent and substantial improvements over Diffusion Policy and ACT, with only a small increase in parameter count and no reduction in inference speed.

Key Points

Privileged 3D supervision uses additional geometric information during training to guide RGB-based visuomotor policy learning, without requiring geometric inputs at deployment.
However, low reconstruction error does not ensure that visual representations capture geometry useful for control.
We identify three limitations that weaken this supervision: proprioceptive shortcuts, dominant-view reliance, and reconstruction objectives dominated by task-irrelevant geometry.
To address these limitations, we propose Latent Encoding with Aligned Privileged Geometry (LEAP)
Our framework uses an auxiliary decoder to reconstruct point clouds within the manipulation workspace from visual features alone, while retaining proprioception for action prediction.

Method

We first identify three limitations of privileged geometric supervision: proprioceptive shortcuts, dominant-view reliance, and reconstruction objectives dominated by task-irrelevant geometry. These issues can allow low reconstruction error without ensuring that visual features capture the geometry needed for control.

Guided by this analysis, we introduce LEAP, which aims to make geometric supervision an observation-aligned and policy-relevant constraint on visual representations.

Method — LEAP: Making Privileged Geometry Supervision Effective for Visuomotor Learning — Figure 2: Preliminary diagnostics of geometry learning. (a) Reconstruction error under unperturbed inputs (Clean), mean replacement (Mean), or shuffling (Shuffle) of robot state $S$ or visual features $Z$. (b) Mean 3D object-position error on two tasks, comparing DP Encoder, DP Encoder with Multi-view setting, and LEAP Encoder with Multi-view setting. Error bars denote standard deviations.

Experimental Results

We evaluate LEAP on 50 RoboTwin tasks, six ManiSkill tasks, and three real-world manipulation tasks. To examine performance across different geometric requirements, we group the RoboTwin tasks into four categories: Object Spatial (14), Multi-object (14), Interaction (14), and Less-geometry (8). We report success rates for each category and the overall average across all 50 tasks.

The ManiSkill and real-world experiments provide complementary evaluations across simulation environments and physical manipulation settings.

Experimental Results — LEAP: Making Privileged Geometry Supervision Effective for Visuomotor Learning — Figure 3: Overview of LEAP. (a) The workflow of an RGB-based policy. (b) During training, LEAP supervises the visual encoder through full and partial point-cloud reconstruction, using visual features alone. Partial reconstruction retains the head and one randomly selected wrist, with targets matched to that wrist’s approximate visibility. The geometry decoder is removed at inference.

Sources and Demonstrations

Figures are reproduced from Fang et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.