SimplerEnv: Simulated Manipulation Policy Evaluation with Real-World Video Prompts

SimplerEnv fills a critical gap in robot learning: how do you rigorously evaluate a real-world manipulation policy without deploying it on expensive physical robots? Introduced at RSS 2024, the work proposes a simulation-based evaluation framework where policies are tested in high-fidelity simulated scenes that are initialized from real-world video observations. By prompting the simulator with a single real-world video frame, the policy sees a distribution of test-time visual conditions that closely mirrors the target deployment environment. The approach leverages state-of-the-art 3D reconstruction (NeRF, Gaussian Splatting, or depth-based mesh generation) to convert real-world video sequences into interactive simulation scenes. Given a video of a kitchen counter, for example, SimplerEnv reconstructs the scene geometry, object shapes, and approximate material properties. It then places the policy's embodiment into the reconstructed scene and evaluates success on the target task — all without any physical robot interaction. The benchmark includes 50+ tasks spanning drawer opening, object relocation, tool use, and articulated object manipulation. Each task provides a real-world video prompt, a reconstructed simulator scene, and standardized evaluation metrics. The authors show strong correlation between SimplerEnv scores and real-world success rates across multiple policies (ACT, Diffusion Policy, RT-1, Octo), validating the approach as a proxy for expensive physical evaluation. A key finding is that adding even minor visual mismatches between the prompt video and the simulator scene significantly degrades policy performance, highlighting the brittleness of current vision-based policies. SimplerEnv also supports counterfactual evaluation: by editing the reconstructed scene (e.g., changing an object's color, adding distractors, or modifying lighting), researchers can perform controlled ablation studies that are impossible in the real world. The codebase is built on top of SAPIEN and ManiSkill, with well-documented scene reconstruction pipelines and evaluation scripts. For any lab that cannot afford extensive real-world robot deployments, SimplerEnv offers a principled, validated alternative for policy benchmarking and iterative development.