Same Trajectory, Contradictory Rewards (ROBORMBENCH): Paraphrase Fragility in Vision Language Reward Models

Wonje Jeung, Sangyeon Yoon, Hyesoo Hong, Yoonjun Cho, Dongjae Jeon, Bumjun Kim, Jean Oh, Youngjae Yu, Albert No · N/A · 2026

Summary

Abstract Summary

Vision-language models are increasingly used as reward functions for robotic learning, but this role requires paraphrase invariance: the same trajectory should receive the same reward under semantically equivalent goal descriptions. We show that current VLM reward models often violate this property. Paraphrasing the instruction alone can substantially change predicted progress scores, and can even flip identical robot behavior between failure and success. To measure this failure mode, we introduce ROBORMBENCH, a benchmark with 2,390 real-robot trajectories, ground-truth progress labels, and 21,673 verified paraphrases spanning lexical, syntactic, and action-goal rewrites. Across proprietary and open-source VLMs, paraphrase-induced instability is widespread and severe, grows under more divergent rewrites, and is not reliably reduced by scale or explicit reasoning. Dedicated reward models trained with trajectory-grounded supervision are substantially more stable. These results show that paraphrase robustness is a core requirement for reliable VLM-based reward modeling in robotics.

Key Points

Vision-language models are increasingly used as reward functions for robotic learning, but this role requires paraphrase invariance: the same trajectory should receive the same reward under semantically equivalent goal descriptions.
We show that current VLM reward models often violate this property
Paraphrasing the instruction alone can substantially change predicted progress scores, and can even flip identical robot behavior between failure and success.
To measure this failure mode, we introduce ROBORMBENCH, a benchmark with 2,390 real-robot trajectories, ground-truth progress labels, and 21,673 verified paraphrases spanning lexical, syntactic, and action-goal rewrites.
Across proprietary and open-source VLMs, paraphrase-induced instability is widespread and severe, grows under more divergent rewrites, and is not reliably reduced by scale or explicit reasoning.

RoboRMBench

We introduce RoboRMBench to evaluate whether language-conditioned reward models assign consistent scores to the same robot trajectory under semantically equivalent goal descriptions. The benchmark pairs each trajectory with linguistically diverse paraphrases of the same goal, allowing us to measure reward instability induced by instruction wording while keeping the visual evidence and task intent fixed.

We consider a language-conditioned reward model that assigns a progress score to a robot trajectory given a textual goal description.

RoboRMBench — Same Trajectory, Contradictory Rewards (ROBORMBENCH): Paraphrase Fragility in Vision Language Reward Models — (a) LS

Experiments

We evaluate RoboRMBench on proprietary and open-source vision-language models spanning multiple scales and capability levels, including both reasoning and non-reasoning variants. For proprietary models, we include GPT-4o, GPT-5.1, Gemini2.5-flash-lite, Gemini2.5-flash, Gemini3-flash, Claude-haiku-4.5, and Claude-sonnet-4.6.

For open-source models, we evaluate the Qwen3-vl, Gemma3, and Llama4 families at multiple sizes. We additionally include dedicated reward models from Lee et al. (2026) : RoboReward-4B (RR-4B) and RoboReward-8B (RR-8B).

Experiments — Same Trajectory, Contradictory Rewards (ROBORMBENCH): Paraphrase Fragility in Vision Language Reward Models — (b) SR

Sources and Demonstrations

Figures are reproduced from Jeung et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.