ManiSkill: A Unified Benchmark for Generalizable Manipulation Skills

Jiayuan Gu, Sean Xiang, Stone Tao, Xuanlin Li, Chongkai Gao, Yihe Tang, Xiaodi Yuan, Yunfan Jiang, Pu Hua, Tongzhou Mu, Fanbo Xiang, Yuzhe Qin, Xiangfeng Hu, Xiaowen Ou, Yen-Yu Chang, Xuanlin Li, Hao Su · UC San Diego, BIGAI · 2023

Summary

ManiSkill is a GPU-parallelized robotics simulation benchmark with 20+ manipulation tasks, PartNet-Mobility assets, and unified observation/action spaces, supporting RL, IL, and VLA training at millions of steps per hour.

Abstract Summary

ManiSkill addresses the fragmentation in robotics benchmarks by providing a single, GPU-parallelized platform for generalizable manipulation. Built on SAPIEN (a high-fidelity physics engine with GPU-accelerated contact handling), ManiSkill supports 20+ tasks including articulated object manipulation (doors, drawers, faucets), tool use (hammering, sweeping), and deformable object handling (cloth folding). All tasks share a unified observation space (RGB-D + segmentation + point cloud) and action space (end-effector pose or joint velocity), making it easy to train and compare policies across tasks. The key technical contribution is GPU batching: ManiSkill runs thousands of parallel environments on a single GPU, achieving 1M+ environment steps per second on an RTX 3090. This is 100x faster than MuJoCo-based benchmarks and enables large-scale RL training (PPO, SAC) and imitation learning (ACT, Diffusion Policy) in hours instead of days. The platform also includes pre-generated demonstration datasets (100K+ trajectories per task) collected with motion planning, enabling offline IL training without real robots. ManiSkill 2.0 (2024) adds support for soft-body physics (cloth, rope, fluids), mobile manipulation (fetching with a mobile base), and humanoid whole-body control. The benchmark has been adopted by the community for evaluating VLA models: OpenVLA, π0, and Octo all report ManiSkill scores as standard metrics. The evaluation protocol includes generalization tests across object instances, poses, and lighting conditions, ensuring policies learn semantic skills rather than memorization. For practitioners, ManiSkill is the fastest way to prototype a manipulation policy: `pip install mani_skill`, load a task, and start training with Stable Baselines3 or your own IL pipeline. The visualizers (3D web viewer + recording) make debugging intuitive.

Key Points

GPU-parallelized simulation: 1M+ steps/sec, 1000+ parallel envs.
20+ tasks: articulated objects, tools, deformable materials.
Unified observation/action spaces across all tasks.
100K+ motion-planned demonstrations for offline IL.
Standard benchmark for VLA models (OpenVLA, π0, Octo).

Building Environments for Generalizable Manipulation Skills

Building high-quality environments demands cross-disciplinary knowledge and expertise, including physical simulation, rendering, robotics, machine learning, software engineering, etc . Our workflow highlights a verification-driven iterative development process, which is illustrated in Appendix A.1 .

Real-time Soft Body Simulation and Rendering

In this section, we describe our new physical simulator for soft bodies and their dynamic interactions with the existing rigid-body simulation. Our key contributions include: 1) a highly efficient GPU MPM simulator; 2) an effective 2-way dynamic coupling method to support interactions between our soft-body simulator and any rigid-body simulator, in our case, SAPIEN.

These features enable us to create the first real-time MPM-based soft-body manipulation environment with 2-way coupling.

Real-time Soft Body Simulation and Rendering — ManiSkill: A Unified Benchmark for Generalizable Manipulation Skills — Figure from the paper. Source: ManiSkill: A Unified Benchmark for Generalizable Manipulation Skills.

Parallelizing Physical Simulation and Rendering

ManiSkill2 aims to be a general and user-friendly framework, with low barriers for customization and minimal limitations. Therefore, we choose Python as our scripting language to model environments, and the open-source, highly flexible SAPIEN ( Xiang et al., 2020 ) as our physical simulator.

The Python language determines that the only way to effectively parallelize execution is through multi-process, which is integrated in common visual RL training libraries ( Raffin et al., 2021 ; Weng et al., 2021 ) .

Parallelizing Physical Simulation and Rendering — ManiSkill: A Unified Benchmark for Generalizable Manipulation Skills — Figure 4: Left: Simulation and real world setup for PickCube . Right: Motion planning execution results for Pinch in simulation and in the real world: (a) initial state; (b) letter “M”; (c) letter “S”.

Applications

In this section, we show how ManiSkill2 supports widespread applications, including sense-plan-act frameworks and imitation / reinforcement learning. In this section, for tasks that have asset variations, we report results on training objects. Results on held-out objects are presented in Appendix F .

Besides, we demonstrate that policies trained in ManiSkill2 have the potential to be directly deployed in the real world. Sense-plan-act (SPA) is a classical framework in robotics.

Sources and Demonstrations

Figures are reproduced from Gu et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.