Diagnosing and Recovering from Observation-Space Shift at Long-Horizon Skill Seams

Summary

Long-horizon robotic manipulation is often built by chaining independently trained skills.

Abstract Summary

Long-horizon robotic manipulation is often built by chaining independently trained skills. Although each skill can be reliable in isolation, performance degrades sharply when skills are chained: each downstream skill must start from the state its predecessor leaves behind rather than from its training distribution. We study this failure mode, Observation-Space Shift (OSS), and ask what causes these skill-seam failures. Using privileged simulator resets, we find that the dominant shift comes from displaced scene state (e.g., an open drawer or secondary objects left behind by earlier skills), not from the robot's joint configuration or the object the downstream skill manipulates. To test this diagnosis, we build a fully learned detect-restore-resume system: a task-progress monitor detects the stall, a learned policy restores the displaced scene components, and seam-robust fine-tuning lets the skill resume. It recovers the seam where every tested alternative fails, which we treat as evidence for the diagnosis rather than as a general-purpose method. On the BOSS-44 benchmark, the system improves full-chain success from 7.6% to 26.5%, a 3.5x improvement over the base policy and 51% of a privileged restoration oracle, whereas best-of-K resampling, a Diffusion Policy, and world-model baselines fail to recover from the evaluated seam states. On a real Franka arm running a fine-tuned $\pi_{0.5}$ policy, the same monitor is limited by exterior-camera observability, yet closing the loop still recovers some otherwise-terminal failures, motivating wrist and gripper sensing. These results suggest that some long-horizon composition failures are better addressed by restoring the scene before resuming the policy than by retrying from an off-support state.

Key Points

Long-horizon robotic manipulation is often built by chaining independently trained skills
Although each skill can be reliable in isolation, performance degrades sharply when skills are chained: each downstream skill must start from the state its predecessor leaves behind rather than from its training distribution.
We study this failure mode, Observation-Space Shift (OSS), and ask what causes these skill-seam failures.
Using privileged simulator resets, we find that the dominant shift comes from displaced scene state (e.g., an open drawer or secondary objects left behind by earlier skills), not from the robot's joint configuration or the object the downstream skill manipulates.
To test this diagnosis, we build a fully learned detect-restore-resume system: a task-progress monitor detects the stall, a learned policy restores the displaced scene components, and seam-robust fine-tuning lets the skill resume.

Problem Formulation

Setting. We study three-skill chains on BOSS-44, a benchmark of $44$ kitchen and living-room manipulation tasks drawn from LIBERO-90 [ 1 ] . Tasks are simulated in robosuite [ 39 ] on the MuJoCo physics engine [ 40 ] , with a $7$-DOF Panda under $20$ Hz OSC delta control. A chain $\mathrm{ch3}\_N$ is an ordered triple of BOSS-44 tasks run in one continuous episode.

For example, $\mathrm{ch3}\_2$ is open the drawer $\to$ place the bowl on the plate $\to$ place the bowl in the cabinet .

Problem Formulation — Diagnosing and Recovering from Observation-Space Shift at Long-Horizon Skill Seams — Fig. 1: Detect, restore, resume at a skill seam. A chained skill collapses when it starts from the previous skill’s off-support terminal state; a task-progress monitor detects the stall, a learned restoration skill returns the displaced scene to support, and the chain resumes.

Method: Detect, Restore, Resume

Recovering a chained skill requires noticing it was stalled, putting the scene back on-support, and letting the skill tolerate an imperfect hand-back. We add one learned component for each, detect , restore , and resume , leaving the base skills frozen. All three read only deployable inputs (RGB cameras and proprioception; Fig. 1 ). Detect answers one question: has the current skill stopped making progress?

A task-progress head scores each frame’s position within a skill, and the trigger fires when that score stops rising (Fig. 3 ).

Method: Detect, Restore, Resume — Diagnosing and Recovering from Observation-Space Shift at Long-Horizon Skill Seams — Fig. 2: The detect–restore–resume loop across scenes and on hardware. Each row runs one chained skill through a seam, left to right. Seam : the downstream skill begins from the previous skill’s off-support terminal state: an open drawer in simulation, a mid-place hand-off on hardware. Detect : the task-progress monitor fires (“Stalled!”). Restore : the learned restoration skill returns the displaced articulation and objects to the skill’s support. Resume : the skill resumes and completes ($\checkmark$). Rows: two catastrophic simulation scenes (S1, open-drawer then place the bowl; S4, a distinct cabinet scene), then hardware chains RC1 (cup$\,\to\,$plate) and RC3 (bowl$\,\to\,$plate). Without the loop, execution halts at the Seam column. Simulation frames are the full system on the paper’s checkpoint (Table IV ); hardware shows recovered place attempts (Sec. V-F ). Each panel boxes the target object in red and shows a magnified inset of it. The box and inset turn green once the object is placed and the skill completes.

Experiments

Metrics. Detection is scored by the AUROC of the alarm statistic over per-seam windows. Overview. We structure the evaluation by first localizing what causes seam collapse and must be restored (Sec. V-A ). We then show that no simpler alternative, like test-time compute, a stronger base policy, or a world model, recovers it (Sec. V-B ). We then present the deployed detect–restore–resume system and its recovery (Sec.

V-C ). Finally, we analyze when restoration is the right action, and why (Sec. V-D ).

Experiments — Diagnosing and Recovering from Observation-Space Shift at Long-Horizon Skill Seams — Fig. 3: The task-progress monitor in action (simulation, S1). Frames A–D from one execution with their monitor scores: reaching ($p{=}0.48$), stalled at the seam ($p{=}0.70$), reset to the seam by restoration ($p{=}0.00$), and completed ($p{=}0.99$). Below, the same scores over time: progress climbs, stalls ( Stall! ), the calibrated trigger fires (dashed), restoration drops it to zero, and the retry resumes. Monitor trigger $+$ learned restoration, no oracle. Phases colored as in Fig. 1 .

Sources and Demonstrations

Figures are reproduced from Wagh et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.