Robotics Research Hub
CALOS: Control-Affine Lyapunov On-manifold Safety Layer for Safe Deep Reinforcement Learning for Quadrotors

CALOS: Control-Affine Lyapunov On-manifold Safety Layer for Safe Deep Reinforcement Learning for Quadrotors

Fabrizio Cesareo, Sebastiano Mengozzi, Nicola Mimmo, Andrea Acquaviva · N/A · 2026

reinforcement-learning control
PDF arXiv

Framework

N/A

License

N/A

Stars

N/A

Summary

Deep Reinforcement Learning has demonstrated remarkable capability in quadrotor control, yet learned policies offer no guarantee of respecting safety constraints during training or deployment.

Abstract Summary

Deep Reinforcement Learning has demonstrated remarkable capability in quadrotor control, yet learned policies offer no guarantee of respecting safety constraints during training or deployment. We present CALOS (Control-Affine Lyapunov On-manifold Safety), a runtime safety layer that enforces attitude constraints on a quadrotor without modifying the underlying learning algorithm. CALOS formulates four tilt-angle inequalities and a Lyapunov descent condition as a single quadratic program whose solution is the minimum-norm correction to the nominal torque output of the policy. The quadratic program is solved exactly via active-set enumeration over the three-dimensional torque space, with a computational cost low enough to enforce constraints in real time across thousands of parallel simulation environments, as required by modern massively parallel Deep Reinforcement Learning training. Evaluated on trajectory-tracking tasks in NVIDIA Isaac Lab, CALOS reduces lateral tracking error by 55-60% relative to an unconstrained Proximal Policy Optimization baseline while achieving zero attitude-constraint violations on the training trajectory. By restricting exploration to safe regions of the state space, the safety layer also accelerates training convergence and improves data efficiency without producing suboptimal policies.

Key Points

  • Deep Reinforcement Learning has demonstrated remarkable capability in quadrotor control, yet learned policies offer no guarantee of respecting safety constraints during training or deployment.
  • We present CALOS (Control-Affine Lyapunov On-manifold Safety), a runtime safety layer that enforces attitude constraints on a quadrotor without modifying the underlying learning algorithm.
  • CALOS formulates four tilt-angle inequalities and a Lyapunov descent condition as a single quadratic program whose solution is the minimum-norm correction to the nominal torque output of the policy.
  • The quadratic program is solved exactly via active-set enumeration over the three-dimensional torque space, with a computational cost low enough to enforce constraints in real time across thousands of parallel simulation environments, as required by modern massively parallel Deep Reinforcement Learning training.
  • Evaluated on trajectory-tracking tasks in NVIDIA Isaac Lab, CALOS reduces lateral tracking error by 55-60% relative to an unconstrained Proximal Policy Optimization baseline while achieving zero attitude-constraint violations on the training trajectory.

Problem Formulation

We consider quadrotor trajectory tracking: a learning-based controller must follow a time-varying reference $p_{d}(t)$ while respecting attitude constraints.

The quadrotor uses the Combined Thrust and Body Torques (CTBT) representation [ 14 ] : the translational and rotational dynamics are: where $m$ is the mass, $v$ the inertial velocity, $R$ the rotation matrix from body to world frame, $F_{d}$ the aerodynamic drag, $J\in\mathbb{R}^{3\times 3}$ the inertia matrix, $\Omega\in\mathbb{R}^{3}$ the body angular velocity, and $\tau\in\mathbb{R}^{3}...

Problem Formulation — CALOS: Control-Affine Lyapunov On-manifold Safety Layer for Safe Deep Reinforcement Learning for Quadrotors
Fig. 1: Circular trajectory $\mathcal{C}$ with large initial offset. Top: $xy$-plane tracking (reference dashed). Bottom: roll and pitch; dashed lines mark the $\pm 60^{\circ}$ constraint. PTP is included to show the effect of the Lyapunov constraint: CALOS-QP settles faster due to active energy dissipation.

Safety Layer Design

Attitude tilt is expressed via the gravity direction in body frame, $g_{b}=R^{\top}e_{3}$, where the subscript $b$ denotes the body-frame representation, avoiding Euler-angle singularities. We impose roll and pitch limits $\phi_{\max}=\theta_{\max}=60^{\circ}$.

Experiments

All agents are trained with PPO [ 13 ] using the DRL library SKRL on 4096 parallel environments in NVIDIA Isaac Lab for ${\sim}10^{9}$ timesteps. The policy is a two-layer MLP 64 units each, with ELU activation and Gaussian output.

The observation $o_{t}\in\mathbb{R}^{15}$ comprises body-frame velocity $v_{b}$, angular velocity $\Omega_{b}$, projected gravity $g_{b}$, desired position $p_{d,b}$, and desired velocity $v_{d,b}$, all in $\mathbb{R}^{3}$.

Sources and Demonstrations

Figures are reproduced from Cesareo et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.

Share

Related Papers