Learning Options for Compositional Motor Control with Adapter Banks

Summary

Abstract Summary

Learning flexible motor primitives is a hallmark of skilled motor control. Recent neuroscience theory proposes that motor primitives may be implemented as low-rank perturbations of a shared recurrent network, but leaves open how such a system is learned. We translate this principle into a novel architecture for learning motor skills end-to-end: a shared recurrent core modulated by a bank of residual adapters, each selected by a discrete latent code. Trained on closed-loop biomechanical control, the adapters develop emergent low-rank perturbations of the recurrent dynamics despite no architectural rank constraint, placing task representations in disparate subspaces of the shared core network. A simple high-level policy over the learned options, optimized while the whole network is frozen, sequences the low-rank adapters to produce novel out-of-distribution movements. We demonstrate the ability to generalize to novel motor sequences within the closed-loop control setting, improving on the generalization error of a task-input-conditioned multitask baseline by upto order of magnitude.

Key Points

Learning flexible motor primitives is a hallmark of skilled motor control
Recent neuroscience theory proposes that motor primitives may be implemented as low-rank perturbations of a shared recurrent network, but leaves open how such a system is learned.
We translate this principle into a novel architecture for learning motor skills end-to-end: a shared recurrent core modulated by a bank of residual adapters, each selected by a discrete latent code.
Trained on closed-loop biomechanical control, the adapters develop emergent low-rank perturbations of the recurrent dynamics despite no architectural rank constraint, placing task representations in disparate subspaces of the shared core network.
A simple high-level policy over the learned options, optimized while the whole network is frozen, sequences the low-rank adapters to produce novel out-of-distribution movements.

Low-rank Adapters via Thalamocortical Architectures in Neuroscience

Theoretical work has proposed an architecture for composing motor primitives into flexible sequences, inspired by the connectivity between motor cortex, thalamus and basal ganglia in the brain [ 12 ] . Here, motor primitives are implemented through an interplay of the recurrence in motor cortex, and cortex-thalamus-cortex projections (thalamocortical loops, Figure 1 A).

Learning Adapters for Flexible Motor Control

To understand how learning system can discover reusable building blocks of behavior that flexibly recombine to solve new problems, we turn to the domain of closed-loop biomechanical motor control. We adopt a closed-loop biomechanical control task from Lazzari and Saxena [18] .

In this task, an RNN is required to control a two-link biomechanical arm with six muscles to generate a variety of movement types: straight reaches, clockwise and counterclockwise circular arcs, and sinusoidal trajectories.

Learning Adapters for Flexible Motor Control — Learning Options for Compositional Motor Control with Adapter Banks — Figure 2: Teacher–student framework for learning motor adapters (A) Task context-conditioned teacher. A recurrent network controls a two-link, six-muscle biomechanical arm in closed loop [ 18 ] . (B) Student encoder. A shared LSTM core network reads the teacher’s full demonstration (observations and muscle commands, but not the rule input) and segments each trajectory into three output probabilities: what task is being done $p(task)$, distribution $p(z)$ over $K=10$ latent codes, and a distribution $p(b)$ of the boundary between the two task segments. (C) Student decoder. The same core network rolls out in closed loop with only body-state feedback and task inputs (no teacher actions). The encoder-sampled code activates the corresponding adapter, which perturbs the cortical hidden state at each timestep.

Results

We first verify that the student successfully reproduces the teacher’s compound movements (Figure 3 , bottom row). Trained only on compound trajectories and without ever being supplied the rule as input, the student traces the expert circles, figure-eights, and reaches with end-to-end hand L1 of approximately order $10^{-2}$ m on all five tasks.

The reach is essentially exact; the figure-eights show closed-loop wobble deviations, but recover the general shape. The more revealing analysis is the forced-code sweep (Figure 3 , top row).

Results — Learning Options for Compositional Motor Control with Adapter Banks — Figure 3: Student execution of compound movements. Top row: Hand-trajectory L1 error of the trained student under all 25 forced (extension code, retraction code) pairs, for each of the five compound tasks. Rows index the five extension codes ($z_{0}$–$z_{4}$); columns index the five retraction codes ($z_{5}$–$z_{9}$). Stars mark the natural code pair selected by the encoder. Bottom row: Expert (black) and student (blue, dashed) hand trajectories under the encoder’s natural code choice. Green dots mark the start position.

Sources and Demonstrations

Figures are reproduced from Kumar et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.