Reflex-Informed Neuromuscular Reinforcement Learning for Muscle-Driven Locomotion

Jian Zhou, Xingyu Zhang, Rui Ma, Yu Cao, Shane Xie, Zhi-qiang Zhang · N/A · 2026

Summary

Abstract Summary

Muscle-driven locomotion provides a physically grounded approach to generating realistic human movement. However, achieving both physiological plausibility and adaptability to changes in musculoskeletal capacity and external disturbances remains a fundamental challenge. To address this limitation, we propose a Reflex-Informed Neuromuscular Reinforcement Learning framework for muscle-driven locomotion. Within this framework, a fixed phase-dependent reflex controller serves as the underlying neuromuscular control mechanism, while the reinforcement learning policy produces four biomechanically meaningful residual parameters to modulate key reflex gains and thresholds associated with hip swing, knee support, and ankle propulsion according to the current state. Experimental results demonstrate that the proposed framework generates physiologically plausible locomotion with improved kinematic accuracy and dynamic consistency, as well as better bilateral symmetry and stride-to-stride consistency under nominal walking conditions. The learned policy remains robust under muscle weakness and external perturbations without retraining.

Key Points

Muscle-driven locomotion provides a physically grounded approach to generating realistic human movement.
However, achieving both physiological plausibility and adaptability to changes in musculoskeletal capacity and external disturbances remains a fundamental challenge.
To address this limitation, we propose a Reflex-Informed Neuromuscular Reinforcement Learning framework for muscle-driven locomotion.
Within this framework, a fixed phase-dependent reflex controller serves as the underlying neuromuscular control mechanism, while the reinforcement learning policy produces four biomechanically meaningful residual parameters to modulate key reflex gains and thresholds associated with hip swing, knee support, and ankle propulsion according to the current state.
Experimental results demonstrate that the proposed framework generates physiologically plausible locomotion with improved kinematic accuracy and dynamic consistency, as well as better bilateral symmetry and stride-to-stride consistency under nominal walking conditions.

Method

Figure 1 illustrates the proposed Reflex-Informed Neuromuscular Reinforcement Learning framework. A fixed phase-dependent reflex controller provides the underlying neuromuscular control mechanism, while the reinforcement learning policy generates four-dimensional residual actions that regulate selected reflex parameters according to the current musculoskeletal state.

The resulting muscle stimulation drives the musculoskeletal simulation, which produces the next state and reward.

Method — Reflex-Informed Neuromuscular Reinforcement Learning for Muscle-Driven Locomotion — Figure 1: Overview of the proposed Residual-Reflex RL framework. (a) The rollout policy maps the musculoskeletal state $\mathbf{s}_{t}$ to a four-dimensional residual action $\mathbf{a}_{t}$ and collects transitions $(\mathbf{s}_{t},\mathbf{a}_{t},r_{t},\mathbf{s}_{t+1})$. (b) The physiological environment combines the muscle-driven character with a phase-dependent residual-reflex-informed controller. In the enlarged lower row, panel (b1) on the left shows the sparse phase-dependent reflex topology, while panel (b2) on the right shows the proprioceptive control loop from muscle sensing to stimulation, activation, muscle–tendon force, skeletal motion, and updated sensory feedback. Magenta entries in (b1) denote the four reflex parameters modulated by $\mathbf{a}_{t}$. Superscripts $(\mathrm{all})$, $(\mathrm{st})$, and $(\mathrm{sw})$ identify parameters shared across all five phases, Early Stance–Late Stance, and Liftoff–Swing–Landing, respectively. The notation $i\leftarrow j$ indicates that feedback from source muscle $j$ modulates target muscle $i$; entries without a source are self-feedback pathways. (c) The collected transitions are stored in $\mathcal{D}$; the critic $q(\mathbf{s},\mathbf{a};\boldsymbol{\omega})$ evaluates candidate actions and MPO updates the rollout policy from $\pi(\mathbf{s};\boldsymbol{\theta}_{\mathrm{now}})$ to $\pi(\mathbf{s};\boldsymbol{\theta}_{\mathrm{new}})$.

Evaluation

The simulator, musculoskeletal character, muscle-driven dynamics, and foot–ground contact model are described in Sec. 3.2 . The same character and simulation settings were used for all controllers unless a plantarflexor weakness condition was explicitly introduced. The evaluation includes four control approaches: two direct muscle-control reinforcement learning methods, E2E-RL Weng et al.

(2021) ; de Vree and Carloni (2021) ; Ogum et al. (2024) and DEP-RL Schumacher et al.

Evaluation — Reflex-Informed Neuromuscular Reinforcement Learning for Muscle-Driven Locomotion — Figure 2: Muscle-driven character model. (a) Front and sagittal views of the H0918v2j character used in the experiments. (b) Schematic arrangement and primary joint functions of the nine bilateral muscle–tendon groups. The complete character is actuated by 18 Hill-type muscle–tendon units, with one instance of each modeled group on each leg.

Sources and Demonstrations

Figures are reproduced from Zhou et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.

Reflex-Informed Neuromuscular Reinforcement Learning for Muscle-Driven Locomotion

Summary

Abstract Summary

Key Points

Method

Evaluation

Sources and Demonstrations

Related Papers

A Balanced Data Diet: Addressing the Exploration Bottleneck in Mega-Scale RL for Robot Control

A Mode-Unified, Continuous, and Biomimetic Control Strategy of a Robotic Lower Limb Prosthesis

A Unified Kinematic Representation Enables Reusable Biological Joint Moment Estimation

Adaptive Rejection of Unknown Hybrid Sinusoidal Disturbances with Application to Humanoid Walking under Unknown Ground Motion