Robotics Research Hub
TouchTherm: Building Multimodal Digital Twins of Objects for Tactile and Thermal Rendering

TouchTherm: Building Multimodal Digital Twins of Objects for Tactile and Thermal Rendering

Yitao Zhang, Hong Ying, Haoran Guo, Xiaoying Zhou, Guanyu Chen, Chenxi Xiao · N/A · 2026

vision reinforcement-learning control tactile benchmark
PDF arXiv

Framework

N/A

License

N/A

Stars

N/A

Summary

Robotic simulation and virtual reality increasingly require object assets that capture not only visual geometry but also the physical cues underlying tactile and thermal interaction.

Abstract Summary

Robotic simulation and virtual reality increasingly require object assets that capture not only visual geometry but also the physical cues underlying tactile and thermal interaction. Existing 3D datasets and reconstruction methods primarily represent object-scale geometry and visual appearance, overlooking microscale surface structure for high-fidelity haptic rendering and transient temperature dynamics for temperature-aware interaction. We present TouchTherm, a framework for constructing simulation-ready visuo-tactile-thermal object assets from real-world objects. For visual and tactile reconstruction, we combine structured-light scanning with multiview normal maps obtained from photometric stereo. The normal maps are registered to the scanned geometry and transformed into tangent space to recover local micro-height fields for optical tactile rendering, while the coarse mesh handles collision detection. For thermal reconstruction, we capture synchronized multiview infrared videos of natural cooling following controlled heating and reconstruct a physics-regularized dynamic thermal field. Experiments on 20 objects show that the reconstructed micro-height fields preserve dominant surface structures and recover higher-frequency details beyond the coarse geometry, while the thermal fields achieve held-out surface-temperature MAEs of 0.465 degrees C and 0.592 degrees C at 30 s and 45 s, respectively. The resulting tactile assets support synthetic-to-real object recognition from tactile observations, while a glove-based VR system demonstrates spatially and temporally varying thermal feedback. These results highlight the potential of TouchTherm for multimodal sensory simulation and temperature-aware virtual interaction.

Key Points

  • Robotic simulation and virtual reality increasingly require object assets that capture not only visual geometry but also the physical cues underlying tactile and thermal interaction.
  • Existing 3D datasets and reconstruction methods primarily represent object-scale geometry and visual appearance, overlooking microscale surface structure for high-fidelity haptic rendering and transient temperature dynamics for temperature-aware interaction.
  • We present TouchTherm, a framework for constructing simulation-ready visuo-tactile-thermal object assets from real-world objects.
  • For visual and tactile reconstruction, we combine structured-light scanning with multiview normal maps obtained from photometric stereo.
  • The normal maps are registered to the scanned geometry and transformed into tangent space to recover local micro-height fields for optical tactile rendering, while the coarse mesh handles collision detection.

Methodology

Our goal is to convert a real object into a simulator-ready multimodal asset comprising a coarse mesh, tactile micro-height fields, and a dynamic thermal field. As shown in Fig. 2 , the pipeline aligns object scans, multiview normal maps, and infrared videos to a common object frame, then reconstructs tactile microgeometry and the dynamic thermal field in two parallel branches.

The resulting representations are integrated into a unified object asset. According to this pipeline, the following sections describe the acquisition setup (Sec.

Methodology — TouchTherm: Building Multimodal Digital Twins of Objects for Tactile and Thermal Rendering
Fig. 3: Overview of the 20 real objects used in our dataset . The collection spans flat, curved, cylindrical, and free-form objects with diverse materials and surface relief.

Experiments

We evaluate TouchTherm from four perspectives: object asset quality (Sec. IV-A ), tactile rendering fidelity (Sec. IV-B ), thermal prediction accuracy (Sec. IV-C ), and downstream utility in perception and interaction tasks (Sec. IV-D ). We constructed digital twins for 20 real objects spanning flat, curved, cylindrical, and free-form geometries, as shown in Fig. 3 .

These objects exhibit smooth, granular, embossed, and anisotropic surface patterns. As summarized in Fig.

Experiments — TouchTherm: Building Multimodal Digital Twins of Objects for Tactile and Thermal Rendering
Fig. 4: Experimental setup for real-to-simulation tactile evaluation . The real and simulated sensors follow matched approach-and-press procedures.

Sources and Demonstrations

Figures are reproduced from Zhang et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.

Share

Related Papers

Norm2Tex: Augmenting Visuo-Tactile Simulations with Texture
arXiv preprint

Norm2Tex: Augmenting Visuo-Tactile Simulations with Texture

Seongjin Bien, D\'ebora Oliveira Makowski, Roberto Calandra et al.

Large-scale datasets are essential for training generalist robot control policies. Collecting real-world tactile data is costly and time-consuming, motivating the use of tactile simulations. However, current tactile simulators capture only overall contact geometry and miss fine details like texture.

vision reinforcement-learning sim-to-real control tactile benchmark
PDF Advanced
A Sim-to-Real Integration Pipeline for Training and Deployment of Chunk-Based VLA Manipulation Policies
arXiv preprint

A Sim-to-Real Integration Pipeline for Training and Deployment of Chunk-Based VLA Manipulation Policies

Mathilde Kappel, Clémence Grislain, Mohamed Chetouani et al.

Vision-Language-Action (VLA) models have become a prominent paradigm for mapping multimodal inputs, including semantic instructions, visual observations of the scene, and proprioceptive observations, to robot actions.

manipulation vision vla reinforcement-learning sim-to-real control learning-from-demonstration benchmark
PDF Advanced