PHIRL: Aligning Learned Rewards with Task Progress for Inverse Reinforcement Learning

Hang Yu, James Staley, Cheng Xi Tsou, Xiujin Liu, Wenchang Gao, Jindan Huang, Shijie Fang, Zhegong Shangguan, Angelo Cangelosi, Reuben Aronson, Elaine Short · N/A · 2026

Summary

Abstract Summary

Human demonstrations provide dense policy-level information but sometimes lack local precision. Human feedback presents accurate local critiques, but offers sparse evaluations rather than direct policy guidance. We propose Progress-Heuristicized Inverse Reinforcement Learning (PHIRL), a data-efficient framework that learns robust reward functions by jointly leveraging demonstrations and feedback. Specifically, we use progress, a feedback modality that describes cumulative task completion. PHIRL iteratively infers a reward function from demonstrations via inverse reinforcement learning, calculates the learned rewards over the progress-annotated demonstrations, and aligns the rewards with progress annotations over four dimensions. We evaluate PHIRL on real and simulated robot tasks, with additional exploration using a fine-tuned vision-language model to provide progress feedback. Results demonstrate that PHIRL significantly outperforms the baselines, achieving substantially higher environmental return rewards and task success with only twenty percent of demonstrations annotated. Analysis of reward-hacking scenarios demonstrates that PHIRL learned reward functions are reliable against exploitation.

Key Points

Human demonstrations provide dense policy-level information but sometimes lack local precision
Human feedback presents accurate local critiques, but offers sparse evaluations rather than direct policy guidance.
We propose Progress-Heuristicized Inverse Reinforcement Learning (PHIRL), a data-efficient framework that learns robust reward functions by jointly leveraging demonstrations and feedback.
Specifically, we use progress, a feedback modality that describes cumulative task completion
PHIRL iteratively infers a reward function from demonstrations via inverse reinforcement learning, calculates the learned rewards over the progress-annotated demonstrations, and aligns the rewards with progress annotations over four dimensions.

Methodology

Our goal is to effectively learn robust reward functions. As previously mentioned, human demonstrations can be combined with human feedback to improve learning efficiency. Our work is built on the key insight that rewards inferred from a set of demonstrations should be consistent with the feedback on the same set of demonstrations. We use a numerical human feedback form, progress [ 20 ] for reward function alignment.

Progress is a signal that describes the cumulative degree of task completion, ranging from fully incomplete to fully complete.

Methodology — PHIRL: Aligning Learned Rewards with Task Progress for Inverse Reinforcement Learning — Fig. 2: Performance comparison and ablation studies Results in the Lift and PickPlaceCan domains. Error bars represent standard deviation. Note that the standard PHIRL model uses 20% annotation (equivalent to PHIRL-20%). (Top) Average environmental return rewards across all baselines. PHIRL significantly outperforms baselines in both domains, particularly when the task horizon is long or demonstration quality is low (MH). (Middle) Ablation study on the effect of removing individual reward shaping components. Completion shaping is most critical when there are failed demonstrations, while reward shaping is most effective overall. (Bottom) Sensitivity analysis of annotation percentages. PHIRL improves learning significantly with 20% of the demonstrations. The benefits of the offline annotation saturate at 50% if the demonstrations are relatively high quality.

Experiments and Results

In this section, we run PHIRL with four baselines using one task with a real robot and two tasks from Robosuite [ 21 ] to validate PHIRL’s performance and our claimed contributions. In addition, we demonstrated that the reward functions learned by PHIRL are robust to reward hacking in two representative scenarios.

Domains We selected two domains from Robosuite, Lift and PickPlaceCan, and one real robot task, animal-brushing. For the Lift domain, the goal is to reach the block and lift it to a certain height.

Experiments and Results — PHIRL: Aligning Learned Rewards with Task Progress for Inverse Reinforcement Learning — Fig. 3: Real Robot Study Results. The goal is to brush the pom-pom off the stuffed animal. We calculated the number of pom-poms that were brushed off the stuffed animal. PHIRL-online achieves the highest.

Sources and Demonstrations

Figures are reproduced from Yu et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.