Embodied Scene Rearrangement Planning

Summary

Abstract Summary

This paper introduces Embodied Scene Rearrangement Planning (ESRP), a novel task requiring embodied agents to rearrange furniture in 3D scenes to match a target configuration using only egocentric observations and a top-down target layout. Unlike prior rearrangement tasks, ESRP precludes global state access and introduces mutual object occlusions, reflecting the practical constraints of real-world robotic deployment. These factors make aligning partial egocentric observations with the global target layout particularly challenging for long-horizon planning. To facilitate research, we present ESRP-Bench, a comprehensive benchmark built on OmniGibson featuring over 5,400 scene pairs and 8,200 objects. We define three multi-level metrics to evaluate rearrangement quality and provide four baselines: a hierarchical task-and-motion planning method, a vision-language-model-based method, and two learning-based approaches (IL and RL). Experimental results demonstrate that current methods struggle to complete the task efficiently, highlighting ESRP as a challenging frontier for embodied agents in scene understanding and long-horizon task planning. This work serves as a stepping stone toward deploying intelligent agents in real-world scenarios. Project page: https://pie-lab.cn/ESRP/.

Key Points

This paper introduces Embodied Scene Rearrangement Planning (ESRP), a novel task requiring embodied agents to rearrange furniture in 3D scenes to match a target configuration using only egocentric observations and a top-down target layout.
Unlike prior rearrangement tasks, ESRP precludes global state access and introduces mutual object occlusions, reflecting the practical constraints of real-world robotic deployment.
These factors make aligning partial egocentric observations with the global target layout particularly challenging for long-horizon planning.
To facilitate research, we present ESRP-Bench, a comprehensive benchmark built on OmniGibson featuring over 5,400 scene pairs and 8,200 objects.
We define three multi-level metrics to evaluate rearrangement quality and provide four baselines: a hierarchical task-and-motion planning method, a vision-language-model-based method, and two learning-based approaches (IL and RL).

ESRP-Bench

esrp defines a task in which an agent needs to reconfigure an unseen 3D indoor scene to match a target layout specified by a single top-down image. Relying solely on first-person views, the agent is required to: (i) comprehend the current scene state through egocentric observations, (ii) formulate an efficient rearrangement strategy, and (iii) execute a sequence of actions to precisely reposition furniture items.

The task is deemed successful only when all rearrangeable objects are correctly placed at their designated goal positions.

ESRP-Bench — Embodied Scene Rearrangement Planning — Fig. 2 : Initial-target scene layout pairs. We group scene layouts into three difficulty levels (as divided by green lines), with the first row showing the initial layout and the second row displaying the corresponding target layout. The top left shows easy-level scenes, where only $1$ furniture object needs to be rearranged; the top right presents medium-level scenes with $2\sim 3$ objects to be rearranged; and the bottom shows hard-level scenes with $4\sim 6$ objects to be rearranged. The dataset covers a wide range of room types, including bedrooms, study rooms, and living rooms, along with diverse furniture types, such as chairs, tables, and plants.

Experiments

Implementation Details For the rl model, we employ ppo with a learning rate of $1.5\times 10^{-4}$ and a value function loss coefficient of $5\times 10^{-3}$. Training is performed with a batch size of $2048$, using minibatches of $64$ for gradient updates. The history context length $l$ is fixed at $64$ time steps. We train the policy for $6$ million environment steps until convergence.

Sources and Demonstrations

Figures are reproduced from Chen et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.