Rapid Learning of Dexterous In-Hand Pen Writing through Real-Time Jacobian Estimation

Rapid Learning of Dexterous In-Hand Pen Writing through Real-Time Jacobian Estimation

|Authors: Kai Stewart, Yasunori Toshimitsu, Robert K. Katzschmann

|Venue: arXiv preprint | Year: 2026

|arXiv: 2609.11775v1

Abstract

Dexterous in-hand manipulation of a grasped object with an anthropomorphic hand is an unsolved frontier for robot dexterity. The contact-richness and highly dynamic nature of object-hand interactions tend to require extensive modeling or data-collection efforts for learning-based approaches. Modern simulators used for reinforcement learning (RL) cannot fully replicate the required contact complexity, while collecting dexterous demonstrations for imitation learning (IL) remains an open problem. In this research, we present an embodied control approach based on real-time task Jacobian estimation of the combined hand and object system on the physical robot. Using only the CPU on a laptop, the proposed controller begins in-hand pen writing after approximately 18 s of initialization and continues to adapt online, without an analytic hand—object kinematic/contact model, simulation training, or precollected task demonstrations. We demonstrate that the same estimator/controller formulation works on three anthropomorphic robotic hand systems (one physical, two simulated) to show human-like, in-hand articulation of a grasped pen by an embodiment-independent formulation. Sub-millimeter in-plane precision (mean 0.6 mm across runs) is achieved across letters and shapes written in the air and on paper on a physical robot. To our knowledge, this is the first demonstration of an anthropomorphic hand writing arbitrary single-stroke trajectories with a grasped pen through purely in-hand motion, and it showcases an alternative to compute- and data-heavy approaches such as RL and IL for achieving dexterous manipulation through computationally simple and data-efficient algorithms.

Key Contributions

• Dexterous in-hand manipulation of a grasped object with an anthropomorphic hand is an unsolved fr…
• The contact-richness and highly dynamic nature of object-hand interactions tend to require extens…
• Modern simulators used for reinforcement learning (RL) cannot fully replicate the required contac…
• In this research, we present an embodied control approach based on real-time task Jacobian estima…
• Using only the CPU on a laptop, the proposed controller begins in-hand pen writing after approxim…

Topics

• manipulation
• reinforcement-learning
• control
• learning-from-demonstration

Code & Data

• GitHub repository: https://github.com/srl-ethz/dexterity_from_jacobian

BibTeX

@article{Stewart2026_260911775v1,
  title     = {Rapid Learning of Dexterous In-Hand Pen Writing through Real-Time Jacobian Estimation},
  author    = {Kai Stewart and Yasunori Toshimitsu and Robert K. Katzschmann},
  year      = {2026},
  eprint    = {2609.11775v1},
  archivePrefix = {arXiv},
  primaryClass  = {cs.RO},
  url       = {https://arxiv.org/abs/2609.11775v1}
}