Asking Smarter Questions: Targeted Active Preference Learning for Underspecified Rewards

Problem Setup

We address the problem of learning a reward function from human input when the teaching signal leaves some features underspecified. Recent work shows that robots can detect underspecified features from statistical patterns in the teaching data [ 3 ] . We use this detection and ask: how should the robot resolve underspecified features?

We propose using that knowledge in two ways: to correct the imperfect posterior into an informed prior, and to direct human attention during preference queries.

Method

Our approach reuses the robot’s diagnosis of which features past teaching left underspecified to elicit targeted preference feedback. First, we construct an informed prior $P(w\mid\mathbf{o})$ that corrects $P(w\mid\mathcal{D})$ block-wise, concentrating uncertainty on $\phi^{u}$.

Second, we pair each query with an explanation that focuses the human’s attention on $\phi^{u}$ when answering, so their responses inform those features.

Simulated Experiments

We evaluate our method in simulation, comparing the informed prior against four alternative priors, a targeted query objective, random querying, and three attention-directing explanation variants. We use two scenes wherein a 7-DoF Jaco arm carries an object above a cluttered surface ( Fig. 2 a–b).

In Stove , the robot carries a utensil across a workspace with a stove, a human, an overhead rack, and four distractors (sponge, banana, soup can, sugar box).

User Study

We conducted a user study to evaluate whether generating queries under the informed prior and providing explanations to guide responses improves reward learning. We also examine whether the explanations improve participants’ subjective experience along three dimensions: ease of answering queries, understanding of their purpose, and expression of what the robot needs to learn.

We conducted an IRB-approved study using the Coffee environment. The 2$\times$2 within-subjects design crossed the prior (uninformed vs.

Real Robot Examples

We use a Franka Emika FR3 arm carrying a knife in a kitchen ( Fig. 2 c). We model the kitchen in PyBullet to generate the trajectory bank and run reward learning offline. The task-relevant features are stove ($xy$-distance to cooktop), cat ($xy$-distance to cat), and point (how directly the knife’s extension axis points at the cat), alongside three distractors: $xy$-distances to a mug , lego , and rubik’s cube .

The ground-truth reward keeps the knife clear of both the stove and the cat and avoids pointing the blade at the cat.

Sources and Demonstrations

Figures are reproduced from Merker et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.