SafeManip: A Property-Driven Benchmark for Temporal Safety Evaluation in Robotic Manipulation

Abstract

Robotic manipulation is typically evaluated by task success, but successful completion does not guarantee safe execution. Many rl failures are temporal: a robot may touch a clean surface after contamination or release an object before it is fully inside an enclosure. We introduce SafeManip, a property-driven benchmark to explicitly evaluate temporal rl properties in robotic manipulation, moving beyond prior simulations that largely focus on task completion or per-state constraint violations. SafeManip defines reusable rl templates over finite executions using Linear Temporal Logic over finite traces (LTLf). It maps observed rollouts to symbolic predicate traces and evaluates them with LTLf-based monitors. Its property suite covers eight manipulation rl categories: collision and contact rl, grasp stability, release stability, cross-contamination, action onset, mechanism recovery, object containment, and enclosure access. Templates can be instantiated with task-specific objects, fixtures, regions, or skills, allowing the same rl specifications to generalize across tasks and environments. We evaluate SafeManip on six vision-language-action policies, including $π_0$, $π_{0.5}$, GR00T, and their training variants, across 50 RoboCasa365 household tasks. Results show that even strong models often behave unsafely. Task-success gains do not reliably translate into safer execution: many successful rollouts remain unsafe, while longer-horizon or more complex tasks expose more violations. SafeManip provides a reusable simulation layer for diagnosing temporal rl failures and measuring safe success beyond task completion.

Links

• Paper (PDF): 2605.12386
• arXiv: 2605.12386