Robotics Research Hub
Synthetic Data Generation and Vision-based Wrinkle and Keypoint Detection for Bimanual Cloth Manipulation

Synthetic Data Generation and Vision-based Wrinkle and Keypoint Detection for Bimanual Cloth Manipulation

Ariel Herrera, Xueyang Kang, Atal Anil Kumar · N/A · 2026

vision manipulation reinforcement-learning
PDF arXiv

Framework

N/A

License

N/A

Stars

N/A

Summary

Robotic manipulation of textiles remains challenging because continuous deformation and self-occlusions hinder the robust visual perception required to estimate the cloth's state.

Abstract Summary

Robotic manipulation of textiles remains challenging because continuous deformation and self-occlusions hinder the robust visual perception required to estimate the cloth's state. To address the lack of annotated real-world data, we developed a Blender-based synthetic pipeline exporting auto-annotated keypoints, and combined manually labeled renders with real-world data to train a wrinkle detector. We present a perception framework integrating a CNN for permutation-invariant keypoint detection and a YOLOv8-OpenCV pipeline to extract grasping points from structural wrinkles. A proposed bimanual algorithm uses this system to stretch fully folded garments via wrinkles, transitioning to keypoint-based ironing once corners emerge. The keypoint model achieves a Mean Position Error (MPE) of 1.7615 pixels. The perception system transfers to physical fabrics without fine-tuning, outperforming baselines that fail in high-occlusion states or yield false positives on severe folds.

Key Points

  • Addresses dexterous manipulation challenges
  • Uses simulation or synthetic data for training
  • Detects fine-grained deformable object features

Method

To train the neural networks without manual annotation, we implemented an automated pipeline using Blender [ 10 ] . We generated a large-scale dataset to ensure robust Domain Randomization. The cloth simulation in Blender is governed by a mass-spring physics model based on Newton’s second law. In our pipeline, the garment is dropped from a random height.

Upon impact with the table collision plane, the solver computes the reaction forces, causing the mesh to buckle and generate highly randomized, realistic deformations and folds.

Method — Synthetic Data Generation and Vision-based Wrinkle and Keypoint Detection for Bimanual Cloth Manipulation
Fig. 1: Examples from the synthetic dataset generated in Blender. (a) Top row: RGB renders showcasing domain randomization, including varying textures, lighting, and complex fold topologies. (b) Bottom row: Corresponding automated ground-truth annotations of the garment’s corner keypoints.

Experiments and Results

We evaluated the keypoint detection model by calculating the Mean Position Error (MPE) between the predicted 2D pixel coordinates and the ground truth from our synthetic test set. Fig. 3 shows the loss and evaluation metrics across all training epochs. Both training and validation loss converged steadily, showing no signs of overfitting.

Furthermore, the validation MPE and PCK@8 (Percentage of Correct Keypoints) stabilized quickly, confirming an effective training process. The model achieves a final MPE of 1.7615 $\pm$ 0.7461 pixels.

Experiments and Results — Synthetic Data Generation and Vision-based Wrinkle and Keypoint Detection for Bimanual Cloth Manipulation
Fig. 2: Conceptual bimanual manipulation sequence simulated using the Newton physics engine. Orange arrows indicate the planned trajectories for the robot end-effectors. (a) Wrinkle-based grasping and stretching for a completely folded garment. (b) Transition to keypoint-based ironing once collinear corners emerge. (c) The final fully unfolded state.

Sources and Demonstrations

Figures are reproduced from Herrera et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.

Share

Related Papers

A Sim-to-Real Integration Pipeline for Training and Deployment of Chunk-Based VLA Manipulation Policies
arXiv preprint

A Sim-to-Real Integration Pipeline for Training and Deployment of Chunk-Based VLA Manipulation Policies

Mathilde Kappel, Clémence Grislain, Mohamed Chetouani et al.

Vision-Language-Action (VLA) models have become a prominent paradigm for mapping multimodal inputs, including semantic instructions, visual observations of the scene, and proprioceptive observations, to robot actions.

manipulation vision vla reinforcement-learning sim-to-real control learning-from-demonstration benchmark
PDF Advanced