An Analysis of Streaming Deep Reinforcement Learning for Adaptive Continual Learning in Robotics

Teeratham Vitchutripop, Alyssa Quarles, Wenhe Zhang, Richard Xue, Daniel Rakita · N/A · 2026

Summary

Abstract Summary

Over the course of a lifetime, robots may encounter novel scenarios unaccounted for in its original training that result in performance degradation. One common approach to mitigating this issue is to further grow the offline training dataset in hopes of producing a policy robust to these changes. In contrast, biological learning occurs moment-to-moment via a stream of experience, unlike the predominantly batch-based and offline nature of deep learning. Although recent works show the feasibility of stream-based deep reinforcement learning, where updates use only the latest experience, none have shown it to be a viable continual learning framework for adapting robotic policies to unseen changes. In this paper, we present the first analysis of streaming deep reinforcement learning for adaptive continual learning in robotics. In particular, we show that, following an initial pretraining phase, streaming deep RL can enable a robot to successfully adapt to unforeseen changes to itself, its environment, or goals. Our primary experiments within quadruped locomotion demonstrate that a deep neural network robotic policy with certain optimizers and plasticity loss mitigation techniques can successfully leverage domain task knowledge from its pretraining to quickly adapt online to diverse changes via stream learning, outperforming batch-based on-policy methods and improving task success rates by up to 90% over the pretrained policy. Furthermore, we perform additional evaluations on robotic manipulation tasks to determine if our previous observations extend to different robotic morphologies and scenarios. Our results show that the successes observed in quadruped locomotion can be partially realized in manipulation with stability and performance limitations. We conclude with a discussion on the limitations of our work and its implications for the future of continual robot learning.

Key Points

Over the course of a lifetime, robots may encounter novel scenarios unaccounted for in its original training that result in performance degradation.
One common approach to mitigating this issue is to further grow the offline training dataset in hopes of producing a policy robust to these changes.
In contrast, biological learning occurs moment-to-moment via a stream of experience, unlike the predominantly batch-based and offline nature of deep learning.
Although recent works show the feasibility of stream-based deep reinforcement learning, where updates use only the latest experience, none have shown it to be a viable continual learning framework for adapting robotic policies to unseen changes.
In this paper, we present the first analysis of streaming deep reinforcement learning for adaptive continual learning in robotics.

Analysis on Quadruped Locomotion

Our primary analysis is a series of experiments conducted within quadruped locomotion that 1) demonstrate the potential capabilities of streaming deep RL for enabling continual robotic adaptation and 2) analyze how the choice of certain optimizers or plasticity loss mitigation techniques can affect adaptation performance. For all policy training, PyTorch was used as the machine learning framework.

Analysis on Other Robotic Environments

We conduct a series of experiments in robotic manipulation environments to see whether our previous observations about the performance of streaming deep RL for adaptation in quadruped locomotion hold with a different task space and robotic morphology. Note that we follow the same experimental setup as defined previously in § IV-A .

Our subsequent analyses are performed on two object manipulation tasks with different robot embodiments in ManiSkill3 [ 23 ] .

Analysis on Other Robotic Environments — An Analysis of Streaming Deep Reinforcement Learning for Adaptive Continual Learning in Robotics — Fig. 3: The performance of 4 Stream-AC variants on the Push Cube goal shift adaptation is shown. All methods show early signs of strong recovery, but performance declines over time after peaking.

Sources and Demonstrations

Figures are reproduced from Vitchutripop et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.