Robotics Research Hub
What Stops Recursive Self-Improvement in Robotics? Lessons from 123 Rounds of Agentic Skill Discovery

What Stops Recursive Self-Improvement in Robotics? Lessons from 123 Rounds of Agentic Skill Discovery

Jiaming Wang (National University of Singapore) · N/A · 2026

manipulation vision reinforcement-learning
PDF arXiv

Framework

N/A

License

N/A

Stars

N/A

Summary

Can a robot improve itself the way coding agents now improve software? We built an agentic system to find out.

Abstract Summary

Can a robot improve itself the way coding agents now improve software? We built an agentic system to find out. It watches a robot fail, works out which capability is missing, writes new skills or finds and installs external models, tests every change in simulation, and repeats, with no human writing robot code. We ran it for 123 improvement rounds on household manipulation tasks. This report describes what we learned. The good news is that the agent can discover capabilities on its own: noticing that its targets were out of view, it asked for an active-viewing model, debugged it, and deployed a working search skill. The bad news is that its improvements did not add up. Changes kept passing their tests, yet the target task, putting condiments on the top shelf of a fridge, never succeeded. We found that the agent was rarely the bottleneck. Three things around it were. First, chained perception modules do not understand relations. Segmenters such as SAM 3 find shelves but not "the top shelf", so the agent filled the gap with ever more geometric rules that never converged, when what it needed was a different kind of model. Second, skill chains lock learning onto the first step. Long tasks mostly fail early, so evidence and fixes pile up there, and later skills are rarely reached, tested, or improved. Third, what the agent learns is decided by the harness. The agent optimized exactly what the evaluator measured, including where it was wrong, and weak tests and misleading memory turned activity into a standstill. We distill these lessons into concrete recommendations for building robot systems that improve themselves, each paired with an experiment that could prove it wrong.

Key Points

  • Can a robot improve itself the way coding agents now improve software?
  • We built an agentic system to find out
  • It watches a robot fail, works out which capability is missing, writes new skills or finds and installs external models, tests every change in simulation, and repeats, with no human writing robot code.
  • We ran it for 123 improvement rounds on household manipulation tasks
  • This report describes what we learned

System Overview

1 Collect run task programs in simulation 2 Judge evaluator checks each declared step 3 Diagnose group first failures into gaps 4 Propose agent edits skills or requests a new model 5 Test & accept replay failure, fresh states, held-out task Research install, probe, register an external model code request next round starts from the improved robot Task program composes skills and declares step goals Routing chooses among a skill’s implementations Harness evaluators, tests, memory, robot interface description Models (providers) segmenters, VLMs, g...

System Overview — What Stops Recursive Self-Improvement in Robotics? Lessons from 123 Rounds of Agentic Skill Discovery
Figure 3: Where each rollout first failed, rounds 66–123. Each row is one task and scene; color is the first step that failed. White circles mark task successes. Early in this window failures move around as skills change; in the last twenty rounds the fridge rows freeze at region finding (“the top shelf”) and grasping, and stay there despite continued library changes and new models.

What happened over 123 rounds

Figure 1 summarizes the whole run, and Figure 3 shows, round by round, where each task first failed. The story has three phases. Early on, the loop did what we hoped. The condiments in Fridge start out of view, and nothing in any prompt said so. At first the localizer returned confident points on visible look-alikes, and the agent tried to re-rank them.

What happened over 123 rounds — What Stops Recursive Self-Improvement in Robotics? Lessons from 123 Rounds of Agentic Skill Discovery
Figure 5: (a) Frontier lock. Fraction of rollouts in the three-task phase whose first $k$ declared steps all passed. More than half of fridge rollouts fail the first step and almost none reach the fifth, so steps beyond the frontier generate little evidence and are rarely exercised by gates. (b) Where the human effort went. The 113 harness repairs made by humans, grouped by what they repaired (the round each one was found in is listed in Table 4 ). Most concern what the agents were told, remembered and rewarded for, not the simulator or the replay machinery (§ 7 ).

Why improvements did not add up

Let $\Phi_{t}$ be the robot’s library after round $t$ and let a task be a chain of $m$ declared steps. Write $p_{j}(\Phi)$ for the probability that step $j$ succeeds given that steps $1,\dots,j-1$ did, and $\pi_{j}=\prod_{i reaches step $j$. Task success is $S(\Phi)=\prod_{j=1}^{m}p_{j}$, which requires no independence assumption.

Each round, the loop observes failures, a proposer produces a change $\delta$, and a gate $G$ accepts it, so that $\Phi_{t+1}=\Phi_{t}\oplus\delta$ when $G(\delta)=1$.

Lesson 1: chained perception modules do not understand relations

The fridge instruction reads: place the condiments on the top shelf; if items already occupy the top shelf, move them to other shelves. Every program the orchestrator wrote therefore asks perception for regions defined by relations: “the top shelf inside the fridge”, “the shelf immediately below the top shelf”, “a food item on the top shelf”.

This is exactly the kind of query household instructions are made of, and the loop never learned to answer it.

Lesson 2: skill chains lock learning onto the first step

A household task is not a skill but a chain of them. The fridge task needs roughly fifteen declared steps: find the top shelf, clear it, then twice find a condiment, approach, grasp, carry, find the shelf again, place and release. This section argues that chaining does more than multiply small unreliabilities. It also shapes what the loop can learn , in a way that keeps it stuck at the front of the chain.

Consider a change that raises the reliability of step $j$ by $\Delta p_{j}$ and leaves everything else fixed.

Lesson 3: what the agent learns is decided by the harness

In a self-improvement loop the proposer supplies variation; everything else (the evaluators that say what went wrong, the gates that accept or reject, the memory that carries experience forward) supplies selection. In software the selection side is a test suite. In robotics it is a set of fallible models of physics and meaning, written by people and never complete.

This section argues that the selection side, not the proposer, determined what our loop learned. We identify four mechanisms by which it failed.

What we would do differently

If we started again, we would build the loop differently. Below are our recommendations, grouped by which of the three conditions each one restores. They come from this experiment but were not themselves tested in it, so Table 3 pairs each with an experiment that could prove it wrong. (D1) Attribute failures by capability, not by location.

An evaluator should ask whether a step could have succeeded from the state in which it began, and assign the failure to the step that made it impossible.

Sources and Demonstrations

Figures are reproduced from Singapore) See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.

Share

Related Papers

A Sim-to-Real Integration Pipeline for Training and Deployment of Chunk-Based VLA Manipulation Policies
arXiv preprint

A Sim-to-Real Integration Pipeline for Training and Deployment of Chunk-Based VLA Manipulation Policies

Mathilde Kappel, Clémence Grislain, Mohamed Chetouani et al.

Vision-Language-Action (VLA) models have become a prominent paradigm for mapping multimodal inputs, including semantic instructions, visual observations of the scene, and proprioceptive observations, to robot actions.

manipulation vision vla reinforcement-learning sim-to-real control learning-from-demonstration benchmark
PDF Advanced