Robotics Research Hub
Norm2Tex: Augmenting Visuo-Tactile Simulations with Texture

Norm2Tex: Augmenting Visuo-Tactile Simulations with Texture

Seongjin Bien, D\'ebora Oliveira Makowski, Roberto Calandra, Florian Walter, Wolfram Burgard · N/A · 2026

vision reinforcement-learning sim-to-real control tactile benchmark
PDF arXiv

Framework

N/A

License

N/A

Stars

N/A

Summary

Large-scale datasets are essential for training generalist robot control policies. Collecting real-world tactile data is costly and time-consuming, motivating the use of tactile simulations. However, current tactile simulators capture only overall contact geometry and miss fine details like texture.

Abstract Summary

Large-scale datasets are essential for training generalist robot control policies. Collecting real-world tactile data is costly and time-consuming, motivating the use of tactile simulations. However, current tactile simulators capture only overall contact geometry and miss fine details like texture. This results in a significant domain shift between simulated and real tactile data. To address this gap, we introduce Norm2Tex, a plug-in method that augments simulations of vision-based tactile sensors with high-frequency surface details from normal map textures. By modifying the target object's depth map before a tactile simulator's rendering pipeline, Norm2Tex seamlessly integrates into different tactile simulators. We also evaluate sim-to-real transfer using material classification and a reinforcement learning task. Our results show that Norm2Tex preserves material-dependent tactile information across domains, improving texture recognition and producing material-dependent control behavior in the real world.

Key Points

  • Large-scale datasets are essential for training generalist robot control policies
  • Collecting real-world tactile data is costly and time-consuming, motivating the use of tactile simulations.
  • However, current tactile simulators capture only overall contact geometry and miss fine details like texture.
  • This results in a significant domain shift between simulated and real tactile data
  • To address this gap, we introduce Norm2Tex, a plug-in method that augments simulations of vision-based tactile sensors with high-frequency surface details from normal map textures.

Method

A normal map is first calculated using a mesh’s original, high-resolution vertices. This map is then applied to a simplified, lower-resolution version of the mesh [ 26 ] . This makes the simpler mesh look more detailed by mimicking how a detailed mesh reflects light. This process saves computational resources during runtime. A UV normal map represents the tangent-space unit normals of the corresponding mesh.

Method — Norm2Tex: Augmenting Visuo-Tactile Simulations with Texture
Fig. 2: The Norm2Tex architecture. First, a procedural texture is created for a given mesh using Blender. At initialization, the global texture height map $H_{\text{global}}$ is computed, and a copy of the mesh is created internally for reference. During simulation, given the relative transform of the mesh to the sensor at the contact point, the corresponding UV coordinates are used to fetch the pixels from $H_{\text{global}}$ to create $H_{\text{local}}$. Finally, $H_{\text{sensor}}$ and $H_{\text{local}}$ are added together to create the final depth image $H_{\text{final}}$, which is then used for rendering the tactile image by the tactile simulator.

Experiments

We perform three separate experiments to validate the usefulness of Norm2Tex. The first experiment assesses its applicability to two existing tactile simulators, TACTO and Taxim, and how it impacts their performance. The second experiment measures sim-to-real capability through a zero-shot classification task.

The final experiment similarly assesses the sim-to-real capability of a tactile policy trained on Norm2Tex data. To assess the performance of Norm2Tex, we implemented our method for two popular tactile simulations, Taxim and TACTO.

Experiments — Norm2Tex: Augmenting Visuo-Tactile Simulations with Texture
Fig. 3: Examples of different fabric renderings generated from a single procedural texture ( fabric ) by adjusting its parameters.

Sources and Demonstrations

Figures are reproduced from Bien et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.

Share

Related Papers

A Sim-to-Real Integration Pipeline for Training and Deployment of Chunk-Based VLA Manipulation Policies
arXiv preprint

A Sim-to-Real Integration Pipeline for Training and Deployment of Chunk-Based VLA Manipulation Policies

Mathilde Kappel, Clémence Grislain, Mohamed Chetouani et al.

Vision-Language-Action (VLA) models have become a prominent paradigm for mapping multimodal inputs, including semantic instructions, visual observations of the scene, and proprioceptive observations, to robot actions.

manipulation vision vla reinforcement-learning sim-to-real control learning-from-demonstration benchmark
PDF Advanced