Robotics Research Hub
Robot Manipulation with GPT-6-Astra: Body Knowledge, Experience Reuse, Emergent Skills, and Sim2Real Transfer

Robot Manipulation with GPT-6-Astra: Body Knowledge, Experience Reuse, Emergent Skills, and Sim2Real Transfer

Sida He, Lingxi Xie, Yunning Cao, Pengfei Chen, Kaiwen Duan, Jiannan Ge, Xinyue Huo, Jiacheng Shao, Qi Tian · N/A · 2026

manipulation vision control learning-from-demonstration
PDF arXiv Code (GitHub)

Framework

N/A

License

N/A

Stars

N/A

Summary

General-purpose multimodal agents can write robot-control programs, but repeated exploration and model-mediated action selection can make execution slow.

Abstract Summary

General-purpose multimodal agents can write robot-control programs, but repeated exploration and model-mediated action selection can make execution slow. We study how external body knowledge, successful experience, and executable skills improve an XLeRobot controlled by GPT-6-Astra in a simulated and a physical elevator-button task. In 30 fixed-start simulation trials, complete robot geometry and camera information reduce mean completion time by 57.4% relative to a baseline with only the common control interface and no prior experience; images with synchronized action and state records reduce it by 68.6% without additional body assets. In nine paired comparisons (18 trials) at starts displaced by 10-100 cm, experience recorded at the original start reduces mean time by 58-63% relative to no experience, demonstrating generalization to the tested new starting positions. During experience experiments, GPT-6-Astra spontaneously generates a short visual-feedback program. Researcher-refactored versions reduce mean local-task time by 29-31% in 27 simulation trials. Finally, 12 real-robot trials using operator-confirmed button contact demonstrate sim2real reuse: at a shared nominal start, simulation XML assets and simulation experience reduce mean time by 53.0% and 49.9%, respectively; real experience also transfers to two new starts. These results suggest a practical way to build general-purpose manipulation experiments around GPT-6-Astra: supply machine-readable body descriptions and synchronized demonstrations, and turn useful agent-generated feedback routines into reusable skills, while the agent adapts actions from current images. We release all task prompts, trial-level experimental data, and acquired skill implementations at https://github.com/hesd10/astra-robot-sim2real.

Key Points

  • General-purpose multimodal agents can write robot-control programs, but repeated exploration and model-mediated action selection can make execution slow.
  • We study how external body knowledge, successful experience, and executable skills improve an XLeRobot controlled by GPT-6-Astra in a simulated and a physical elevator-button task.
  • In 30 fixed-start simulation trials, complete robot geometry and camera information reduce mean completion time by 57.4% relative to a baseline with only the common control interface and no prior experience; images with synchronized action and state records reduce it by 68.6% without additional body assets.
  • In nine paired comparisons (18 trials) at starts displaced by 10-100 cm, experience recorded at the original start reduces mean time by 58-63% relative to no experience, demonstrating generalization to the tested new starting positions.
  • During experience experiments, GPT-6-Astra spontaneously generates a short visual-feedback program

System and Experimental Protocol

The simulated and physical platforms are based on XLeRobot [ 18 ] , with two six-channel arms, a pan–tilt head, and three onboard cameras. We modified the original platform to use a four-wheel mecanum base, enabling forward, lateral, and rotational motion. The MuJoCo [ 19 ] environment reproduces the modified robot’s geometry and control channels in an elevator lobby (Figure 3 ).

System and Experimental Protocol — Robot Manipulation with GPT-6-Astra: Body Knowledge, Experience Reuse, Emergent Skills, and Sim2Real Transfer
Figure 3: The MuJoCo elevator lobby used in the simulation experiments. (a) The cutaway view shows the robot at the original starting state, facing three elevators. (b) The enlarged panel shows the upward call button beside the middle elevator. The agent observes the scene through its head and wrist cameras.

Body Knowledge and Experience in Simulation

All 30 fixed-start trials succeed. Without historical experience, mean task times for I0, I1, I2, and I3 are 1024.9, 1160.4, 579.8, and 437.1 s (Figure 4 (a)). Thus mechanical assets alone reduce mean time by 43.4%, and complete geometry with camera information reduces it by 57.4% relative to I0.

The qualitative I1 condition is not faster, demonstrating that adding information does not automatically reduce execution cost. At I0, mean times for E0, E1, E2, and E3 are 1024.9, 878.1, 850.7, and 321.4 s (Figure 4 (b)).

Body Knowledge and Experience in Simulation — Robot Manipulation with GPT-6-Astra: Body Knowledge, Experience Reuse, Emergent Skills, and Sim2Real Transfer
Figure 4: Fixed-start simulation results: (a) body-information levels without experience; (b) experience representations at the minimum body-information level. Bars show means; dots show all three trials per condition. Geometry and synchronized action experience have the clearest gains. The full table also includes I3 experience conditions.

From Emergent Behavior to Reusable Skills

During a supplementary simulation study comparing experience packages (Appendix C.3 ), Astra writes a program that advances briefly, captures images, checks red pixels within selected regions for button activation, and repeats until a threshold is reached or the step budget is exhausted.

The program permits at most five iterations; the observed episode executes four, taking about 4.6 s locally, followed by Astra’s visual confirmation.

From Emergent Behavior to Reusable Skills — Robot Manipulation with GPT-6-Astra: Body Knowledge, Experience Reuse, Emergent Skills, and Sim2Real Transfer
Figure 5: Reusing a working right-arm pose in simulation. (a) Initial configuration. (b) The selected experience source at 481 s, after it has found the arm pose used for pressing. (c) A subsequent I0E3 trial at 141 s, after adopting the same key joint targets. The shoulder-pitch, elbow, and wrist-pitch targets are $(q_{2},q_{3},q_{4})=(1.0,1.2,-0.84)$ rad in (b,c). Views are rendered from recorded simulation configurations using the same camera angle relative to the robot. The agent continues to adjust the base from current observations.

Sim2Real and Real-Experience Transfer

We evaluate the physical XLeRobot in a real elevator lobby after the calibration and interface preparation described in Section 3 . Hardware calibration expresses joint targets in the same XML reference as simulation, and both APIs use body-frame base velocities. The task changes from the middle elevator’s UP button to the opposite-side DOWN button, with success determined by operator-confirmed gripper-tip contact.

The controller allows velocity updates during continuous base motion and stops when a command expires or a stop is requested.

Sim2Real and Real-Experience Transfer — Robot Manipulation with GPT-6-Astra: Body Knowledge, Experience Reuse, Emergent Skills, and Sim2Real Transfer
Figure 6: A complete simulated approach using recorded experience, completed in 310 s. (a) Initial position. (b) Advancing with an intermediate arm configuration. (c) Extending into the working pose. (d) Approaching the middle elevator’s panel. (e) Final alignment before activation. (f) The gripper activates the upward call button; its indicator turns red. Timestamps use the task clock. Frames are rendered from recorded states with a fixed external camera focused on the robot, middle elevator, and panel; insets in (e,f) enlarge the button interaction.

Sources and Demonstrations

Figures are reproduced from He et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.

Share

Related Papers

A Sim-to-Real Integration Pipeline for Training and Deployment of Chunk-Based VLA Manipulation Policies
arXiv preprint

A Sim-to-Real Integration Pipeline for Training and Deployment of Chunk-Based VLA Manipulation Policies

Mathilde Kappel, Clémence Grislain, Mohamed Chetouani et al.

Vision-Language-Action (VLA) models have become a prominent paradigm for mapping multimodal inputs, including semantic instructions, visual observations of the scene, and proprioceptive observations, to robot actions.

manipulation vision vla reinforcement-learning sim-to-real control learning-from-demonstration benchmark
PDF Advanced
Capability-Aware Arbitration for Semantic Intent-Based Shared Control
arXiv preprint

Capability-Aware Arbitration for Semantic Intent-Based Shared Control

Zhaoda Du, Michael Bowman, Xiaoli Zhang

Shared control often allocates robot authority based on confidence in inferred human intent, assuming reliable autonomous execution. When this assumption fails, high intent confidence can cause over-helping.

manipulation vision vla reinforcement-learning control learning-from-demonstration benchmark
Code PDF Intermediate
GitHub ★ —
DexVerse: A Modular Benchmark for Multi-Task, Multi-Embodiment Dexterous Manipulation
arXiv preprint

DexVerse: A Modular Benchmark for Multi-Task, Multi-Embodiment Dexterous Manipulation

Yunchao Yao, Zhuxiu Xu, Tianqi Zhang et al.

Building general-purpose dexterous manipulation policies requires benchmarks that go beyond isolated tasks to systematically evaluate policies across diverse interaction modes, sensory conditions, and robot embodiments.

manipulation vision vla reinforcement-learning control learning-from-demonstration benchmark
PDF Advanced