KiRoSe: Using Kinematic Data for Efficient Segmentation of Robot Demonstrations

At a glance

Problem: annotating subtask structure in robot demonstrations by hand is tedious and error-prone, and feeding long videos to multimodal models is expensive. Source
Idea: train a kinematic encoder with subtask-transition supervision, quantize its pooled features into Sparse Motion Tokens, and teach a language model to read them. Source
Evidence: on 200 held-out YAM episodes, boundary F1 within 1 second is 0.63, versus about 0.41–0.42 for alternative kinematic tokenizers using the same reader. Source

Evaluation scope Everything trainable is learned from 5,478 episodes of the YAM bimanual robot; evaluation uses 200 held-out YAM episodes, 2,653 AgiBot World episodes and 193 human-hand trajectories from Ego-Exo4D. Source

How it works

Robots and human hands share one 14-dimensional end-effector representation of left and right linear velocity, angular velocity and gripper opening rate. Source
A temporal encoder is trained with an auxiliary head that predicts transition probabilities at each time step, supervised by annotated subtask transitions. Source
The frozen encoder's features are pooled on a uniform time grid and vector-quantized into discrete motion tokens, each carrying its timestamp. Source
The language model reads the task instruction together with the motion-token sequence, with textual time markers inserted every second as a reference for segment boundaries. Source
The language model is supervised with the reference segmentation serialized as JSON segments, each with a start time, an end time and a natural-language description. Source

KiRoSe pipeline: a transition-supervised encoder turns the kinematic stream into Sparse Motion Tokens, and a language model reads them with time markers and the instruction to output timed action descriptions. Original figure

Evidence & results

Transferred zero-shot to AgiBot, KiRoSe reaches 0.40 boundary F1 at ±0.5 s, compared with 0.48 on held-out YAM and 0.19 for the video-based Qwen3.5-9B baseline. Evidence
On AgiBot, action-level grounded F1 is 0.59 for KiRoSe versus 0.34 for GPT-6 Astra. Evidence
When subtasks are played in reverse, KiRoSe's predicted progress falls on 86% of 2,586 reversed AgiBot subtasks, while ReWiND stays positively correlated. Evidence

Zero-shot reversal probe on AgiBot: one subtask per episode is played backwards, and the curves compare ReWiND's progress, which keeps rising, with KiRoSe's, which falls. Original figure

Limitations & boundaries

Accuracy is lower on human hands than on robots, and the method has not been evaluated on long-horizon mobile manipulation. Source
The Ego-Exo4D vocabulary result is qualitative, and the authors attribute the supplied words to the language model's existing knowledge of the task name rather than semantics learned from the training data. Source
In the reversal test KiRoSe's forward progress correlation is +0.51, lower than ReWiND's +0.88 and Robometer-4B's +0.71. Source

Takeaway

Our reading: kinematics look like a cheap, transferable signal for finding when manipulation actions change, but the authors advise spot-checking automatic labels before using them to train policies. Source

Resources

Full text ↗PDF ↗

Figures: the paper's authors. Editorial source and image review: 2026-10-26. Findings are author-reported, not independently reproduced by Robotics Research Hub.

Original abstract

Segmenting robot demonstration data into disjoint temporal and semantic subsegments is valuable for learning, cleaning, retrieval, and analysis. Segmentation is often performed manually in a tedious and error prone process. Multimodal Large Language Models (MLLMs) have the potential for automating this process, but directly ingesting raw multi camera video streams into MLLMs can be costly and can quickly exhaust context windows. In this paper, we introduce KiRoSe (Kinematics for Robot Segmentation), a novel framework that focuses on the companion stream of robot kinematic data. Inspired by the principle of classical entropy coding, such as Huffman coding, where frequently occurring symbols are represented more compactly, we propose Sparse Motion Tokens (SMTs), a discrete kinematic vocabulary designed to compactly represent recurring, temporally localized motion patterns, which we refer to as kinematic events. SMTs are learned by encoding kinematic trajectories with temporal segment supervision and quantizing the resulting features. We use increasingly available robot demonstration subtask annotations to supervise an open language model to interpret the resulting token sequences. Evaluation against human labeled segmentation shows that KiRoSe can substantially outperform baseline segmentation methods in F1 quality and transfers zero shot to unseen robot embodiments and egocentric human motion, while preserving the language models open vocabulary knowledge. Beyond segmentation, KiRoSe tracks task progress on an unseen robot and detects when an action is being reversed.(website: https://berkeleyautomation.github.io/kirose/)

Citation

@article{Xie2026_261031801,
  title     = {KiRoSe: Using Kinematic Data for Efficient Segmentation of Robot Demonstrations},
  author    = {Shuangyu Xie and Kaiyuan Chen and Muhammad Zubair Irshad and Richard Cheng and Sergey Zakharov and Ken Goldberg},
  year      = {2026},
  eprint    = {2610.31801},
  archivePrefix = {arXiv},
  primaryClass  = {cs.RO},
  url       = {https://arxiv.org/abs/2610.31801}
}