ORDER: A Fictitious-World Benchmark for Domain-Adaptive Embodied AI

Summary

Abstract Summary

Adapting language models to new domains via continual pre-training raises a basic evaluation problem: if the training corpus overlaps with what the model already knows, performance gains cannot be cleanly attributed to new learning rather than pre-existing knowledge. This matters most for knowledge-intensive, task-light (KHTL) robot deployments - pharmaceutical dispensing, hazardous-material handling, facility-specific protocols, where the physical task is simple but the governing rules are proprietary and safety-critical, and where extensive live testing is costly or unsafe. We introduce ORDER (Ontology-driven Decision-making for Embodied Reasoning), a benchmark built on a fictitious world: a 342,069-token synthetic corpus defining a self-consistent physics that cannot appear in any model's pre-training data. ORDER pairs a 500-question knowledge test (ORDER-BENCH) with a harder compositional task, ORDER-SPATIAL: ordering objects for safe manipulation across both familiar and entirely novel scenes. GPT-4.1 without adaptation scores below chance on ORDER-SPATIAL (Kendall's tau = 0.441), showing its priors actively conflict with the invented physics. After continual pre-training, small models improve substantially on both familiar and novel scenes alike evidence of genuine world-model induction rather than memorization. We then carry this through to a robot pipeline: models that answer the knowledge test well often cannot produce valid, executable plans without a further skill-adaptation stage, after which small, fully offline models outperform GPT-4.1 even when GPT-4.1 is given retrieval access to the same rules (Kendall's tau = 0.848 vs. 0.606), on a full perception-to-execution loop demonstrated on a simulated iiwa7 arm with human-in-the-loop correction. Throughout, ORDER-SPATIAL performance, not knowledge-test accuracy is what predicts real plan quality.

Key Points

Adapting language models to new domains via continual pre-training raises a basic evaluation problem: if the training corpus overlaps with what the model already knows, performance gains cannot be cleanly attributed to new learning rather than pre-existing knowledge.
This matters most for knowledge-intensive, task-light (KHTL) robot deployments - pharmaceutical dispensing, hazardous-material handling, facility-specific protocols, where the physical task is simple but the governing rules are proprietary and safety-critical, and where extensive live testing is costly or unsafe.
We introduce ORDER (Ontology-driven Decision-making for Embodied Reasoning), a benchmark built on a fictitious world: a 342,069-token synthetic corpus defining a self-consistent physics that cannot appear in any model's pre-training data.
ORDER pairs a 500-question knowledge test (ORDER-BENCH) with a harder compositional task, ORDER-SPATIAL: ordering objects for safe manipulation across both familiar and entirely novel scenes.
GPT-4.1 without adaptation scores below chance on ORDER-SPATIAL (Kendall's tau = 0.441), showing its priors actively conflict with the invented physics.

The Order Framework

(0) Master Ontology contract $\cdot$ primitives intrinsics $\cdot$ pairwise higher-order regimes (1) Axioms conservation laws dominance rules (2) Counterfactuals one violated constraint per entry (4) Spatial Configs 70 CPT scenes with analysis, $+$ $\pi^{*}$ (3) QnA $\cdot$ (5) Paraphrases same knowledge, varied surface form Order-World corpus CPT input ( text field) Base SLM Mistral-7B LLaMA-3.1-8B Qwen3-4B CPT via LoRA knowledge adapter (K-LoRA) on $\mathcal{D}$ Task Arithmetic Merge K-LoRA $+$ instruct $\Rightarrow$ KI-SLM Skill SFT (S-LoRA)...

Order-Bench, Order-Spatial, and Evaluation Metrics

Order-Bench comprises 500 four-option multiple-choice questions (MCQ) measuring knowledge extraction fidelity : how completely and accurately a model can recover and apply the domain physics from $\mathcal{D}$. Every question has a gold answer directly recoverable from corpus text; compositional generalisation to novel configurations is evaluated separately by Order-Spatial .

Questions span seven categories derived from every corpus layer (Table II ); per-category distractor design is specified in Appendix F-B .

Knowledge Ingestion Variants

Seven K-LoRA variants are evaluated across three SLM families (Table III ). All use LoRA [ 41 ] on all linear layers ($r\!=\!64$, $\alpha\!=\!128$, dropout $0$, lr $=5\!\times\!10^{-5}$, 10 epochs, effective batch size 4) and are merged onto the instruct model via task arithmetic [ 42 ] ($\alpha_{1}\!=\!\alpha_{2}\!=\!0.5$). Full algorithmic specifications appear in Appendix E .

Benchmark Experiments

We evaluate Mistral-7B-Instruct-v0.3 [ 45 ] (M-7B), LLaMA-3.1-8B-Instruct [ 46 ] (L-8B), and Qwen3-4B-Instruct-2507 [ 47 ] (Q-4B), a hybrid reasoning architecture with thinking mode enabled by default. 1 1 1 Thinking mode in Qwen3 reflects realistic deployment for hybrid-reasoning SLMs; see Appendix M for the evaluation-fairness discussion.

GPT-4.1 serves as the frontier reference in both closed-book and retrieval settings. Full PPL profiles and per-section MCQ breakdowns for all K-LoRA variants are in Appendices C and G .

From Knowledge to Action: System Architecture

The remainder of the paper carries the validated models from benchmark scores to executable plans on a manipulator. The system decouples perception, cognition, and execution across three layers. This decoupling is deliberate: it isolates visual understanding from symbolic planning, allows independent evaluation of each layer, and makes domain knowledge swappable without re-training the perception or execution modules.

A lightweight off-the-shelf VLM, SigLIP2 [ 48 ] , serves as the scene analyser.

Knowledge–Skill Separation and Skill Fine-Tuning

A natural starting hypothesis is that a model with strong Order-Spatial performance might zero-shot generate executable BTs encoding those orderings. We test this directly before any S-LoRA training, evaluating all KI variants on 82 novel test scenes under the same XML prompt used for S-LoRA evaluation. Table VIII reports results.

Across Mistral-7B and LLaMA-3.1-8B, svr is essentially zero for nearly all KI-adapted variants.

Retrieval for Fine-Tuned SLMs: Context Hijacking and Two-Stage Recovery

Given that retrieval nearly saturates Order-Bench MCQ (Section VI-B ), a natural question is whether the same structured retrieval index can augment fine-tuned SLMs at BT-generation time. The answer, in its unadapted form, is no, and the reason is architectural: context hijacking . The structured retrieval index (Appendix A-B ) assembles rich analytical physics descriptions within a 4,000-token budget.

Human-in-the-Loop Execution on iiwa7 in Gazebo

Experiments are conducted in Gazebo (ROS 1 Noetic) using an open-source iiwa7 arm model. The tabletop workspace contains primitively coloured geometric objects (cubes, cylinders, spheres, cones, discs) in colours and textures drawn from the five-axis visual grammar $\mathcal{G}$, enabling direct correspondence between the Gazebo scene and the Order-World evaluation world.

Object poses are published via /scene_perception as structured JSON $\{O,C,P,R\}$.

Closing the Knowledge-to-Action Chain

Table XIII compares top-performing Domain-BT-LM variants against GPT-4.1 references. All Domain-BT-LM configurations are closed-book : deployed without any retrieval at inference. All variants shown exceed the best GPT-4.1 configuration ($\tau_{\mathrm{order}}=0.606$) by margins of $0.190$–$0.242$, and the $\tau^{u}_{\mathrm{order}}$ column makes the knowledge-to-action chain legible per family. LLaMA.

Sources and Demonstrations

Figures are reproduced from Sathi et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.