PrefO2O: Zero-Shot Preference-Based Reinforcement Learning via Offline-to-Online Latent Preference Transfer

At a glance

Problem: preference-based RL usually needs fresh human preference labels for every new task, which limits real-world deployment. Source
Idea: learn a task-invariant latent preference space offline from several source tasks, then use it on the target task for reward priors and optimal-transport pseudo-labels. Source
Evidence: after sim-to-real transfer to a physical Kinova Jaco2 arm, PrefO2O succeeded in 7/10 cube-stacking and 6/10 pick-and-place trials. Source

Evaluation scope Source-task supervision is small: 100 human preference annotations per task, from graduate students and staff, over pairs sampled from a mixed-quality trajectory pool. Source

How it works

Offline, a trajectory encoder, a preference encoder and a meta reward model are trained jointly on source-task preference data. Source
Online, the pretrained trajectory encoder and reward model warm-start the target task while the preference encoder stays frozen as a reference for optimal-transport alignment. Source
The alignment produces confidence-weighted pseudo-preference labels that drive continued reward adaptation and policy optimization on the target task. Source

PrefO2O overview: the offline phase learns a trajectory encoder, preference VAE and meta reward model from source tasks; the online phase warm-starts the target task and aligns preferences in the latent space. Original figure

Evidence & results

Moving from intra- to cross-embodiment transfer, PrefO2O drops by 5 to 15 percentage points, while the other preference-transfer baselines' rankings collapse. Evidence
PrefO2O has the highest pseudo-label accuracy on all benchmarks in the intra-embodiment setting, and keeps most of it across embodiments where PEARL and RL-VLM-F collapse. Evidence
With a single source task, many one-to-one curves stay competitive, although using several source tasks generally performs best. Evidence
On the same physical arm, the PEARL baseline succeeded in 2/10 cube-stacking and 1/10 pick-and-place trials. Evidence

Pseudo-label accuracy across three benchmarks, mean over five random seeds with one-standard-deviation error bars. Original figure

Limitations & boundaries

Transfer to tasks whose preferences depend on behaviors absent from every source task remains an open challenge. Source
Optimal-transport alignment runs on sampled mini-batches for tractability, and scaling it to very large source pools is left for future study. Source
Pseudo-label accuracy is measured against a scripted teacher that labels preferences from ground-truth environment rewards, not against new human judgments. Source

Takeaway

Our reading: transferring preferences through a learned latent space looks more robust than aligning raw trajectories, but the authors limit the claim to related task families that share behavioral structure. Source

Resources

Full text ↗PDF ↗

Figures: the paper's authors. Editorial source and image review: 2026-10-26. Findings are author-reported, not independently reproduced by Robotics Research Hub.

Original abstract

Preference-based reinforcement learning (PbRL) has emerged as a promising paradigm for learning robot behaviors aligned with human intent. However, its scalability is fundamentally limited by the need to collect expensive human preference labels for every new task. In this work, we present PrefO2O, an offline-to-online latent preference transfer framework that enables zero-shot PbRL on unseen target tasks without requiring any human feedback. The core idea is to learn a task-invariant latent preference space from a small amount of offline preference data on source tasks, where transferable preference structures are disentangled from task-specific variations. This learned manifold allows PrefO2O to transfer preference structures and reward priors to new tasks under distribution shift, while confidence-weighted optimal transport alignment generates reliable pseudo-preference supervision for online reward adaptation. Extensive experiments on robotic manipulation and locomotion benchmarks demonstrate that PrefO2O substantially outperforms existing preference-transfer baselines and matches or exceeds PbRL methods trained with real human feedback, while generalizing across multi-to-one and one-to-one transfer settings as well as across different robot embodiments.

Citation

@article{Wang2026_261031812,
  title     = {PrefO2O: Zero-Shot Preference-Based Reinforcement Learning via Offline-to-Online Latent Preference Transfer},
  author    = {Ruiqi Wang and Dezhong Zhao and Xuesu Xiao and Erdem Biyik and Byung-Cheol Min and Guohua Chen},
  year      = {2026},
  eprint    = {2610.31812},
  archivePrefix = {arXiv},
  primaryClass  = {cs.RO},
  url       = {https://arxiv.org/abs/2610.31812}
}