FasTac: A Curved Multispectral Vision-Based Tactile Sensor for High-Speed High-Precision 3D Shape and Force Perception
Xiaofan Lu, Kaiji Huang, Jiahui Chen, Yuankai Lin, Hua Yang, Zhouping Yin · N/A · 2026
Framework
N/A
License
N/A
Stars
N/A
Summary
Curved tactile fingertips for dexterous manipulation must resolve fine contact geometry, distinguish normal and tangential loads, and capture transient signals.
Abstract Summary
Key Points
- Curved tactile fingertips for dexterous manipulation must resolve fine contact geometry, distinguish normal and tangential loads, and capture transient signals.
- Existing curved vision-based tactile sensors struggle to combine accurate 3D reconstruction, three-axis force estimation, and high-speed processing in a compact form.
- This article presents FasTac, a curved vision-based tactile sensor integrating multispectral photometric stereo, dynamic-convolution force estimation, and hardware acceleration on a field-programmable gate array (FPGA).
- Single-image-sensor simultaneous multispectral imaging provides spatially aligned observations for robust surface normal estimation, followed by boundary-prior fast Poisson depth reconstruction.
- HyperForce uses position-aware dynamic convolution to model the spatially nonuniform mechanical response of curved elastomers and estimate three-axis forces.
FasTac Sensor Design
The mechanical design of FasTac prioritizes compactness and integration for dexterous manipulation. As shown in Fig. 2 , the sensor mainly consists of three modules: a light-guiding fingertip skeleton that houses the camera, a multispectral LED FPC secured by the rear cover, and a sensing interface formed by the gel skin. FasTac adopts a custom-designed miniature camera module.
The camera module is controlled by an onboard FPGA, which receives raw data through the MIPI interface.
FasTac 3D Shape Reconstruction and Force Estimation Pipeline
The overall perception pipeline is illustrated in Fig. 4 , including multispectral 3D reconstruction, marker-based tangential displacement extraction, 3D displacement fusion, and HyperForce-based three-axis force estimation. The geometric perception of FasTac is based on photometric stereo [ 24 ] .
For a Lambertian surface with normal $\boldsymbol{n}$ and albedo $\rho$, the observed intensity under illumination direction $\boldsymbol{l}$ is $I=\rho\boldsymbol{l}^{\mathrm{T}}\boldsymbol{n}$.
Experiments
Fig. 6 summarizes the automated acquisition pipeline. A force-sensor-mounted spherical indenter applied controlled normal and tangential contacts while tactile images, CNC poses, and force readings were recorded synchronously. Pixel-wise normal and depth ground truth was rendered from the pose, indenter geometry, and aligned sensor CAD model. Spatially disjoint datasets were used for training, validation, and testing.
Data Collection and Training Protocol
In the setup shown in Fig. 6 of the main paper, FasTac was fixed on a CNC platform, and a spherical indenter mounted on an ATI Nano17 force/torque sensor applied controlled contacts. Tactile images, CNC poses, and force readings were recorded synchronously. Four corner points defined the valid sensing region, within which calibration grids were generated by bilinear interpolation.
At each grid point, vertical contacts were recorded under normal loads of 0.5, 1.0, and 1.5 N.
High-Frequency Vibration Measurement Platform
As shown in Fig. S1 , the frequency controller drives the platform in sinusoidal vertical motion, causing the probe to periodically indent the FasTac surface and produce a dynamic normal-force response.
Edge Deployment and Dynamic Evaluation Details
The CPU and GPU baselines were evaluated on an AMD Ryzen 7 8845H CPU and an NVIDIA RTX 4060 Laptop GPU, respectively, while the FPGA implementation targeted a Xilinx Zynq UltraScale+ MPSoC (XCZU19EG-2FFVC1760I). FasTac was connected through an FPC cable to an FMC-to-MIPI adapter mounted on the FPGA board.
For edge deployment, the original $1920\times 1080$ images used in the offline experiments of the main paper were cropped and downsampled to $128\times 128$.
Sources and Demonstrations
Figures are reproduced from Lu et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.
Related Papers
Design and Evaluation of a Touchscreen-Based Teleoperation Interface for Robotic Manipulators
Juan José García Cárdenas, Alperen Kenan, Hamidreza Raei et al.
Intuitive teleoperation interfaces are crucial for the safe and effective operation of robotic manipulators in challenging environments.
DUET: Decoupled Upper- and Lower-Body Execution for Task-Agnostic Humanoid Loco-Manipulation
Kapalik Khanal, Biswash Khatiwada, Stephen Afrifa et al.
Changing a humanoid robot's manipulation policy should not require retraining the controller that maintains the robot's balance.
FACT: A Multimodal Force-Action-Track Foundation Model for Human-to-Robot Dexterity
Zhi Cao, Yining Hong, Jiaolong Yang et al.
Human demonstrations offer a scalable alternative to costly robot data collection for dexterous manipulation, but most human datasets lack force measurements.
ForeTac-VLA: A Forecasting-Based Tactile-Vision-Language-Action Model for Contact-Rich Robotic Manipulation
Zhengyu Tao, Xin Li, Xin Wang
Vision-language-action (VLA) models have demonstrated strong capabilities in robotic manipulation, yet their reliance on visual perception limits robustness in contact-rich environments, where critical physical interaction states may not be visually observable.