Robotics Research Hub
Teaching a Robot Dog New Tricks: Diverse Quadruped Skills via Combined Reinforcement and Imitation Learning with Adversarial Task Selection

Teaching a Robot Dog New Tricks: Diverse Quadruped Skills via Combined Reinforcement and Imitation Learning with Adversarial Task Selection

Lemon Foxmere, Anthony Furman, Yizheng Du, Oliver Chang, Leilani Gilpin, Steve McGuire · N/A · 2026

locomotion reinforcement-learning learning-from-demonstration benchmark
PDF arXiv

Framework

N/A

License

N/A

Stars

N/A

Summary

Reinforcement Learning (RL) has enabled legged robots to perform a range of skills in single-task settings. However, applications such as farm robotics or space exploration require diverse skills such as locomotion, digging, or close-range surveying.

Abstract Summary

Reinforcement Learning (RL) has enabled legged robots to perform a range of skills in single-task settings. However, applications such as farm robotics or space exploration require diverse skills such as locomotion, digging, or close-range surveying. Training an end-to-end policy to address this problem remains difficult due to challenges such as sample inefficiency and gradient conflict between tasks in multi-task learning. We propose a three-stage method that trains a single policy to perform distinct tasks such as walking, digging, and hopping, and compose them into novel behaviors such as crawling. First, multiple teacher policies are trained using RL on narrowly defined tasks. Then, two additional stages train a student policy with a multi-teacher distillation setup that uses a combined RL and Imitation Learning (IL) objective under an adversarial task selection process that focuses training on the worst-performing task. With this method, we train a student policy that performs 22 tasks using 8 teachers. Evaluations show our method preserves motion quality and tracks commands more accurately than PPO and distill-then-finetune baselines, and in some cases generalizes to new tasks without explicit training. Finally, we demonstrate real-world robustness by deploying the resulting policy on a Unitree B1 quadruped. Video: https://youtu.be/V9yX04EBcFA

Key Points

  • Reinforcement Learning (RL) has enabled legged robots to perform a range of skills in single-task settings.
  • However, applications such as farm robotics or space exploration require diverse skills such as locomotion, digging, or close-range surveying.
  • Training an end-to-end policy to address this problem remains difficult due to challenges such as sample inefficiency and gradient conflict between tasks in multi-task learning.
  • We propose a three-stage method that trains a single policy to perform distinct tasks such as walking, digging, and hopping, and compose them into novel behaviors such as crawling.
  • First, multiple teacher policies are trained using RL on narrowly defined tasks

Methodology

We model the multi-task problem as a Markov Decision Process (MDP) defined by the tuple $(\mathcal{S},\mathcal{A},\mathcal{T},\mathcal{R},\gamma)$, where $\mathcal{S}$ is the state space, $\mathcal{A}$ is the action space, $\mathcal{T}$ is the transition function, $\mathcal{R}$ is the reward function, and $\gamma$ is the discount factor.

Methodology — Teaching a Robot Dog New Tricks: Diverse Quadruped Skills via Combined Reinforcement and Imitation Learning with Adversarial Task Selection
Fig. 2: An overview of our method. Teacher policies are first trained via PPO to perform each single task. During ILP, a student policy is optimized over a combined $L_{\mathrm{PPO}}-\lambda_{\mathrm{IL}}L_{\mathrm{BC}}$ objective, with the behavioral cloning demonstrations collected from the teachers via DAgger. In this stage, rewards only encourage body stability and motion smoothness. During RLP, a new critic is first pre-trained on the RLP reward landscape, and the student is optimized over the same combined objective but with a weaker $\lambda_{\mathrm{RL}}$. Here, rewards are extended to also finetune certain behaviors and encourage command tracking. Throughout both ILP and RLP, adversarial task selection is used during training to accumulate more transitions on the worst performing tasks.

Experiments and Evaluations

We evaluate the performance of our method on 8 single tasks and 14 non-trivial and dynamically feasible composite tasks. Each policy is trained on all 22 tasks, listed and described in Table II . For composite tasks, we only train on 2-task compositions. All compositions of three or more tasks are reserved for generalization evaluation.

We compare our method against three baselines: Distill-then-Finetune (DtF) as used by Rudin et al. [ 15 ] , single-stage PPO, and a teacher router that selects the corresponding teacher policy for a given task.

Real-World Deployment

To achieve robust real-world deployment, we apply standard domain randomization techniques during training, including observation noise and delays. Each observation channel’s noise is modeled as a Gaussian with a bias and variance, where the bias is sampled uniformly from a fixed range at the start of each episode.

We additionally model external force disturbances as forces applied to the robot’s center of mass over a randomized time duration.

Sources and Demonstrations

Figures are reproduced from Foxmere et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.

Share

Related Papers

Has GPT-6-Astra Achieved Embodied Intelligence?
arXiv preprint

Has GPT-6-Astra Achieved Embodied Intelligence?

SenseNova, Joint Research Team, : et al.

GPT-6-Astra demonstrates a level of competence in robotic manipulation and navigation that challenges the conventional paradigm of building specialized models for individual tasks and embodiments.

manipulation locomotion vision reinforcement-learning planning control learning-from-demonstration benchmark
Code PDF Advanced
GitHub ★ —
WholeBodyVLA: Towards Unified Latent VLA for Whole-Body Loco-Manipulation Control
arXiv preprint

WholeBodyVLA: Towards Unified Latent VLA for Whole-Body Loco-Manipulation Control

Haoran Jiang, Jin Chen, Qingwen Bu et al.

Humanoid robots require precise locomotion and dexterous manipulation to perform challenging loco-manipulation tasks. Yet existing approaches, modular or end-to-end, are deficient in manipulation-aware locomotion.

manipulation locomotion vision vla reinforcement-learning control learning-from-demonstration benchmark
Code PDF Advanced
GitHub ★ —