Robotics Research Hub
Digital Twin-Driven VR Teleoperation with Multi-View Spatial Perception for Surgical Robots

Digital Twin-Driven VR Teleoperation with Multi-View Spatial Perception for Surgical Robots

Chang Liu, Chenhao Yu, Honghao Zhao, Hao Ding, Haochen Wei, Adnan Munawar, Mathias Unberath, Peter Kazanzides · N/A · 2026

vision reinforcement-learning control learning-from-demonstration
PDF arXiv Code (GitHub)

Framework

N/A

License

N/A

Stars

N/A

Summary

Current robot-assisted minimally-invasive surgery (RMIS) platforms provide a fixed console for the surgeon to view stereo endoscopic images and teleoperate instruments inside the patient.

Abstract Summary

Current robot-assisted minimally-invasive surgery (RMIS) platforms provide a fixed console for the surgeon to view stereo endoscopic images and teleoperate instruments inside the patient. Several researchers have proposed the use of a head-mounted display (HMD) as a portable console, with video pass-through rendering of the endoscope images which, like the fixed console, restricts the operator to a single endoscopic viewpoint and limits depth perception. We present a digital twin-driven virtual reality (VR) teleoperation platform, where the digital twin is created from markerless perception of the surgical environment and streamed for display on the HMD. This overcomes the limitations of video pass-through by providing multi-view rendering and natural motion-parallax cues, enabling decoupling of the user's hand posture from strict instrument alignment. The system utilizes VR hand controllers to increase the teleoperation workspace and to improve the robustness and stability of instrument control compared to the hand tracking approach adopted by most prior systems. A 15-participant user study on the da Vinci Research Kit (dVRK) shows that our VR platform significantly outperforms a state-of-the-art HoloLens 2 mixed reality baseline, reducing path length by 86% and jerk by 95%, while achieving depth perception confidence comparable to or exceeding the traditional console across all conditions.

Key Points

  • Current robot-assisted minimally-invasive surgery (RMIS) platforms provide a fixed console for the surgeon to view stereo endoscopic images and teleoperate instruments inside the patient.
  • Several researchers have proposed the use of a head-mounted display (HMD) as a portable console, with video pass-through rendering of the endoscope images which, like the fixed console, restricts the operator to a single endoscopic viewpoint and limits depth perception.
  • We present a digital twin-driven virtual reality (VR) teleoperation platform, where the digital twin is created from markerless perception of the surgical environment and streamed for display on the HMD.
  • This overcomes the limitations of video pass-through by providing multi-view rendering and natural motion-parallax cues, enabling decoupling of the user's hand posture from strict instrument alignment.
  • The system utilizes VR hand controllers to increase the teleoperation workspace and to improve the robustness and stability of instrument control compared to the hand tracking approach adopted by most prior systems.

System Overview

Figure 1 summarizes the end-to-end pipeline across the physical layer, computation layer, and user interface.

Our platform consists of three asynchronous flows: (i) Robot state stream: the dVRK publishes high-rate robot joint states as JSON packets over UDP via sawSocketStreamer 1 1 1 https://github.com/jhu-saw/sawSocketStreamer , which are consumed by Unity to update the robot digital twin in real time; (ii) Object state stream: an external RGB-D camera observes the workspace and sends images to a perception server, where segmentation and 6-Do...

System Overview — Digital Twin-Driven VR Teleoperation with Multi-View Spatial Perception for Surgical Robots
Fig. 2: Perception GUI for managing multiple object digital twins.

Experiments

To evaluate the proposed digital twin-driven VR teleoperation platform, a user study was conducted comparing it against two established paradigms: the traditional stationary console (MTM) of the da Vinci Research Kit (dVRK), and the console-free MR teleoperation system of Magnani et al. [ 3 ] , which uses HoloLens 2 with bare-hand tracking and virtual instrument shafts.

The objective was to assess how different platforms influence users’ depth perception and spatial understanding during a fine manipulation task.

Experiments — Digital Twin-Driven VR Teleoperation with Multi-View Spatial Perception for Surgical Robots
Fig. 3: Registration of the digital twin. The physical dVRK (a) and workspace (c) are aligned with their virtual counterparts in Unity (b, d) using a unified camera-centric coordinate frame.

Sources and Demonstrations

Figures are reproduced from Liu et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.

Share

Related Papers

A Sim-to-Real Integration Pipeline for Training and Deployment of Chunk-Based VLA Manipulation Policies
arXiv preprint

A Sim-to-Real Integration Pipeline for Training and Deployment of Chunk-Based VLA Manipulation Policies

Mathilde Kappel, Clémence Grislain, Mohamed Chetouani et al.

Vision-Language-Action (VLA) models have become a prominent paradigm for mapping multimodal inputs, including semantic instructions, visual observations of the scene, and proprioceptive observations, to robot actions.

manipulation vision vla reinforcement-learning sim-to-real control learning-from-demonstration benchmark
PDF Advanced
Capability-Aware Arbitration for Semantic Intent-Based Shared Control
arXiv preprint

Capability-Aware Arbitration for Semantic Intent-Based Shared Control

Zhaoda Du, Michael Bowman, Xiaoli Zhang

Shared control often allocates robot authority based on confidence in inferred human intent, assuming reliable autonomous execution. When this assumption fails, high intent confidence can cause over-helping.

manipulation vision vla reinforcement-learning control learning-from-demonstration benchmark
Code PDF Intermediate
GitHub ★ —