Scale Up Strategically: Learning Compositional Generalization via Bias-Aware Evaluation and Data Collection for Robotic Manipulation

Yu Qi, Zhang Ye, Xinyi Xu, Yuxuan Lu, Amitoj Sandhu, Boce Hu, Haojie Huang, Jonathan Tremblay, Lawson L. S. Wong · N/A · 2026

Summary

Compositional generalization is essential for robot to follow diverse instructions. However, pretrained policies are known to take shortcuts, deferring to salient cues rather than grounding language.

Abstract Summary

Compositional generalization is essential for robot to follow diverse instructions. However, pretrained policies are known to take shortcuts, deferring to salient cues rather than grounding language. We introduce a diagnostic framework that localizes this failure to individual \textit{instruction factors}, \textit{e.g.,} reusable semantic components such as color, verb, object, size, and spatial attribute. Our framework formalizes instruction factor bias, the tendency of fine-tuned policies to over-rely on dominant factors as shortcuts, and quantifies it through two metrics: Factor Dominance Rate (FDR), capturing pairwise bias between factors, and Factor Dominance Hierarchy (FDH), aggregating these into a global ranking. Evaluation on six foundation policies reveals broadly consistent ordering, \textit{i.e.}, color $\geq$ object $\geq$ spatial $\geq$ verb $\geq$ size, with color dominant, and verb and size most under-grounded. We further show the diagnosis is actionable: a bias-aware data collection strategy that reallocates a fixed budget toward under-grounded factors outperforms baselines in simulation and on a real robot using half the demonstrations, thereby enabling more sample-efficient and generalizable policy learning.

Key Points

Compositional generalization is essential for robot to follow diverse instructions
However, pretrained policies are known to take shortcuts, deferring to salient cues rather than grounding language.
We introduce a diagnostic framework that localizes this failure to individual \textit{instruction factors}, \textit{e.g.,} reusable semantic components such as color, verb, object, size, and spatial attribute.
Our framework formalizes instruction factor bias, the tendency of fine-tuned policies to over-rely on dominant factors as shortcuts, and quantifies it through two metrics: Factor Dominance Rate (FDR), capturing pairwise bias between factors, and Factor Dominance Hierarchy (FDH), aggregating these into a global ranking.
Evaluation on six foundation policies reveals broadly consistent ordering, \textit{i.e.}, color $\geq$ object $\geq$ spatial $\geq$ verb $\geq$ size, with color dominant, and verb and size most under-grounded.

Problem Formulation

As shown in Fig. 2 , our experiments are conducted in a multi-task tabletop manipulation setup in ManiSkill [ 33 ] , where each natural-language instruction is decomposed into five factors following the template “ [Verb] the [Size] [Color] [Object] on the [Spatial Attribute] of the table ” (e.g., “Grasp the smallest red cube on the left of the table” ).

Evaluation of Factor Bias

This section introduces our factor-level diagnostic framework. We first define Factor Dominance Rate (FDR) to measure pairwise factor dominance, and then aggregate pairwise scores into Factor Dominance Hierarchy (FDH) to reveal global bias patterns across models. To probe the bias within a factor pair $(f_{1},f_{2})$, we finetune a policy on a training distribution that deliberately correlates the two factors (In Fig.

Strategic Data Collection with Factor Bias

We hypothesis that the consistent factor hierarchy observed in Finding 3 can directly guide data collection. Rather than searching for an optimal sampling strategy globally, our goal is to provide a simple yet effective bias-aware data collection strategy which yields substantial gains in compositional generalization and data efficiency. Fig.

5 visualizes different sampling strategies for 2 factors, axes enumerate the possible values of the two factors, blue dots mark the samples collected, while gray dots mark the held-out samples.

Strategic Data Collection with Factor Bias — Scale Up Strategically: Learning Compositional Generalization via Bias-Aware Evaluation and Data Collection for Robotic Manipulation — Figure 5: Methods.

Experiment

To test whether the consistency revealed in Finding 3 translates into an effective data collection strategy, we conducted experiments across multiple models in both simulation in Sec. 6.1 and real-robot experiments in Sec. 6.2 . Additionally, following the exact setup as Gao et al. [13] . We keep the sum of total factor change $f$ and demonstration budget $n$ the same across methods for fair comparisons.

Sources and Demonstrations

Figures are reproduced from Qi et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.