Same Action, Different Outcome: Variability in Dynamic Cloth Manipulation

Summary

Although cloth is known to exhibit different outcomes under repeated fast dynamic motions, even when the same trajectory is applied, this variability has not yet been systematically characterized.

Abstract Summary

Although cloth is known to exhibit different outcomes under repeated fast dynamic motions, even when the same trajectory is applied, this variability has not yet been systematically characterized. Quantifying it is essential to assess the reliability of learned manipulation policies and the extent to which simulation can reproduce real-world behavior. To study this, we execute the same trajectory ten times across four dynamic tasks, two of which are novel, each tested with three cloths of very different properties and at up to three execution speeds, with a total of 269 recorded rollouts. For all of them, we record small marker positions on the cloth and synchronized stereo camera. We then formalize different metrics to quantify variability, and our results show how it is significant in every test condition, in most cases one to three orders of magnitude above the repeatability inherent to the robot and sensing noise. Our results also show variability is driven mainly by the cloth physical properties but also grows with speed. By replaying all the rollouts in four calibrated modern cloth simulators, we show that none of them can reproduce the variability magnitude we observed in real cloth, nor its ordering. Together with our analysis, we publish the dataset with synchronized OptiTrack, vision and robot logs and their corresponding simulator twins.

Key Points

Although cloth is known to exhibit different outcomes under repeated fast dynamic motions, even when the same trajectory is applied, this variability has not yet been systematically characterized.
Quantifying it is essential to assess the reliability of learned manipulation policies and the extent to which simulation can reproduce real-world behavior.
To study this, we execute the same trajectory ten times across four dynamic tasks, two of which are novel, each tested with three cloths of very different properties and at up to three execution speeds, with a total of 269 recorded rollouts.
For all of them, we record small marker positions on the cloth and synchronized stereo camera
We then formalize different metrics to quantify variability, and our results show how it is significant in every test condition, in most cases one to three orders of magnitude above the repeatability inherent to the robot and sensing noise.

State of the Art

Several examples exist in the literature that prove the advantage of dynamic manipulation. For instance, quasi-static pipelines smooth garments by sequences of pick-and-place actions [ 15 , 16 ] . A fling unfolds a cloth in a fraction of those interactions [ 7 ] , and other dynamic actions followed: flings that align garments [ 8 ] , single-arm folding [ 3 , 4 , 14 ] and placing [ 17 ] .

That nominally identical dynamic executions leave the cloth in different states has been observed but not characterized. Coltraro et al.

State of the Art — Same Action, Different Outcome: Variability in Dynamic Cloth Manipulation — Fig. 2 : (Left) Bimanual FR3 cell with OptiTrack cameras to track retroreflective markers on the cloth while the two arms execute the grasp poses. (Right) Marker distribution.

Problem Formulation

Interaction variability is the non-repeatability of the cloth state under the same trajectory and a nominally identical initial state. Let $\pi$ be the manipulation policy and $y_{t}$, $r_{t}$, $u_{t}$ the observed cloth state, robot state, and control input at time $t$. Here $\pi$ replays an open-loop trajectory, so $u_{t}=\pi(y_{t},r_{t})$ is deterministic.

The next cloth state is where $p_{\mathrm{real}}$ is the probability distribution over the next state given the current observation and control input.

Problem Formulation — Same Action, Different Outcome: Variability in Dynamic Cloth Manipulation — Fig. 3 : Each row shows 5 frames of the recorded tasks. The markers used to study the variability for each task are colored and tracked in the camera view. Other relevant markers are in gray.

Experimental protocol

Two Franka Research 3 arms with default Franka Hands are placed side by side across a table (Fig. 2 -left). An OptiTrack motion-capture system tracks markers at $100\text{\,}\mathrm{Hz}$ in a world frame with $x$ away from the robot base and $z$ up. A stereo camera (ZED 2i) records the bimanual-with-cloth scene at $30\text{\,}\mathrm{Hz}$.

Additionally, four ArUco tags, each with tracking markers placed on the table top-right corner, are placed on the table to provide the transformation matrix between the OptiTrack origin and the camera.

Experimental protocol — Same Action, Different Outcome: Variability in Dynamic Cloth Manipulation — Fig. 4 : We show the variability amount for all tasks and their configurations with the whisker boxes of all the pair distances $D_{c}(i,j)$ in all rollouts per configuration. The variability is orders of magnitude different depending on the task; that is why the y-axis is on a logarithmic scale.

Results

We give an overall idea of the variability amount across the different tasks at the different configurations in Fig. 4 . The graphic shows the whisker boxes for the pair distances $D_{c}(i,j)$ of each of the 27 configurations, and their means $\bar{D}_{c}$ described for each task in the previous section following Eq. 3 and 4 .

The y-axis is logarithmic because $\bar{D}_{c}$ spans from 1.2 mm, corresponding to the brown cloth slow swing join task, up to 118 mm for the folding with the green satin, a factor of 96.

Results — Same Action, Different Outcome: Variability in Dynamic Cloth Manipulation — Fig. 5 : Resting poses of the markers of the bottom edge for the swing place task (Task 1). It shows increasing variability for lighter cloths as well as with higher speeds.

Measuring the dispersion without markers

Motion capture is the most time-consuming part of the data collection because it often requires manual re-labeling of markers that get lost for some frames. In addition, most deployed policies use RGB(D) cameras; therefore, in this section we investigate how much of the same analysis as in Section V-A we can perform without markers, using data from the stereo camera that recorded all our experiments.

Sources and Demonstrations

Figures are reproduced from Dadgostar et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.