Robotics Research Hub
Design and Evaluation of a Touchscreen-Based Teleoperation Interface for Robotic Manipulators

Design and Evaluation of a Touchscreen-Based Teleoperation Interface for Robotic Manipulators

Juan José García Cárdenas, Alperen Kenan, Hamidreza Raei, Paul Bremner, Manuel Giuliani, Arash Ajoudani, Adriana Tapus · N/A · 2026

manipulation vision reinforcement-learning control learning-from-demonstration tactile benchmark
PDF arXiv

Framework

N/A

License

N/A

Stars

N/A

Summary

Intuitive teleoperation interfaces are crucial for the safe and effective operation of robotic manipulators in challenging environments.

Abstract Summary

Intuitive teleoperation interfaces are crucial for the safe and effective operation of robotic manipulators in challenging environments. In the nuclear industry, surface contact tasks such as swab sampling require precise path and force tracking, obstacle avoidance, and sustained operator attention, which conventional joystick interfaces struggle to support effectively. This study designs and evaluates a novel touchscreen teleoperation interface that maps continuous finger movements directly to robotic manipulator motions, provides finer velocity control, and integrates control with visualization, enabling more natural, precise, and intuitive surface interaction than conventional controllers. A comparative user study with 20 participants evaluated task performance and workload using the proposed touchscreen, a conventional joystick, and a single-click autonomous mode. Tasks simulated realistic surface manipulation using a Franka Emika Panda arm, remotely controlled from another country. Kinematic, physiological, and behavioral data were recorded to comprehensively assess task performance, cognitive load, and operator trust across each control condition. Participants completed teleoperation tasks more efficiently and accurately with the touchscreen interface, achieving a 53.5% reduction in completion time (median: 2.50 vs. 5.38 min), higher in-area coverage on the sinusoidal path (90.7% vs. 84.1%), and lower overshoot on both path geometries compared with the joystick. Cognitive load, quantified via NASA-TLX (0-100), decreased from joystick to touchscreen (mean TLX 52 to 43; -9 points, -17.3%) and was lowest under the autonomous one-click mode (31; -21 points vs. joystick, -40.4%; -12 vs. touchscreen, -27.9%). This research presents an easy-to-implement touchscreen interface that improves performance in teleoperated surface tasks while reducing cognitive load.

Key Points

  • Intuitive teleoperation interfaces are crucial for the safe and effective operation of robotic manipulators in challenging environments.
  • In the nuclear industry, surface contact tasks such as swab sampling require precise path and force tracking, obstacle avoidance, and sustained operator attention, which conventional joystick interfaces struggle to support effectively.
  • This study designs and evaluates a novel touchscreen teleoperation interface that maps continuous finger movements directly to robotic manipulator motions, provides finer velocity control, and integrates control with visualization, enabling more natural, precise, and intuitive surface interaction than conventional controllers.
  • A comparative user study with 20 participants evaluated task performance and workload using the proposed touchscreen, a conventional joystick, and a single-click autonomous mode.
  • Tasks simulated realistic surface manipulation using a Franka Emika Panda arm, remotely controlled from another country.

Methodology

To evaluate the proposed touchscreen-based interface for robot teleoperation, a comparative user study was conducted against an industry-standard joystick controller. The objective was to assess usability, cognitive load, and task performance. Tasks were modelled on realistic scenarios such as swabbing in nuclear maintenance, designed to replicate precise, contact-based manipulation over a surface.

Figure 2 displays a full-screen screenshot of the actual interface used by the participants.

Methodology — Design and Evaluation of a Touchscreen-Based Teleoperation Interface for Robotic Manipulators
Fig. 2 : Interface display captured during trial execution

Teleoperation Control Implementation

The communication architecture for the teleoperation setup is illustrated in Figure 4 , which involves four PCs located in different countries. In Bristol, UK, PC1 served as the human-robot interface, capturing finger position data and displaying the robot’s camera feed, while PC2 logged experimental data.

In Genoa, Italy, PC3 converted position reference data into a time-synchronised ROS topic, and PC4 acted as the ROS Master, controlling the Franka Emika Panda arm and enforcing safety constraints.

Teleoperation Control Implementation — Design and Evaluation of a Touchscreen-Based Teleoperation Interface for Robotic Manipulators
(a) Joystick controller

Hypotheses

Building on research in collaborative HRI, three different hypotheses are formulated: Participants in condition C3 (Supervisory one-click autonomy) will have the lowest cognitive load in comparison with the other two conditions of teleoperation. Participants in condition C2 (Touchscreen interface) will elicit significantly lower workload and higher trust ratings than the baseline condition C1 (Joystick interface).

Hypotheses — Design and Evaluation of a Touchscreen-Based Teleoperation Interface for Robotic Manipulators
(b) Touchscreen controller

Results

All 20 operators finished both manual conditions C1 and C2. In terms of completion time, the touchscreen condition (C2) had a median of 2.50 min compared to 5.38 min with the joystick condition (C1), representing a 53.5% reduction.

Consistent with this, the ECDF (Empirical Cumulative Distribution Function) plot in Figure 5 shows that the curve for C2 rises more steeply, corresponding to shorter completion times, whereas the curve for C1 is flatter and shifted to the right, indicating slower completion times.

Results — Design and Evaluation of a Touchscreen-Based Teleoperation Interface for Robotic Manipulators
Fig. 4 : Overview of the communication framework

Sources and Demonstrations

Figures are reproduced from Cárdenas et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.

Share

Related Papers

A Sim-to-Real Integration Pipeline for Training and Deployment of Chunk-Based VLA Manipulation Policies
arXiv preprint

A Sim-to-Real Integration Pipeline for Training and Deployment of Chunk-Based VLA Manipulation Policies

Mathilde Kappel, Clémence Grislain, Mohamed Chetouani et al.

Vision-Language-Action (VLA) models have become a prominent paradigm for mapping multimodal inputs, including semantic instructions, visual observations of the scene, and proprioceptive observations, to robot actions.

manipulation vision vla reinforcement-learning sim-to-real control learning-from-demonstration benchmark
PDF Advanced
Capability-Aware Arbitration for Semantic Intent-Based Shared Control
arXiv preprint

Capability-Aware Arbitration for Semantic Intent-Based Shared Control

Zhaoda Du, Michael Bowman, Xiaoli Zhang

Shared control often allocates robot authority based on confidence in inferred human intent, assuming reliable autonomous execution. When this assumption fails, high intent confidence can cause over-helping.

manipulation vision vla reinforcement-learning control learning-from-demonstration benchmark
Code PDF Intermediate
GitHub ★ —
DexVerse: A Modular Benchmark for Multi-Task, Multi-Embodiment Dexterous Manipulation
arXiv preprint

DexVerse: A Modular Benchmark for Multi-Task, Multi-Embodiment Dexterous Manipulation

Yunchao Yao, Zhuxiu Xu, Tianqi Zhang et al.

Building general-purpose dexterous manipulation policies requires benchmarks that go beyond isolated tasks to systematically evaluate policies across diverse interaction modes, sensory conditions, and robot embodiments.

manipulation vision vla reinforcement-learning control learning-from-demonstration benchmark
PDF Advanced