<p>In online continual learning, a neural network learns from a continuous data stream in which each data point is processed only once and never revisited in the next learning stages. This non-revisiting mechanism often leads to Catastrophic Interference (CI), i.e., the model can dramatically lose the learned features of previous tasks when new information is learned. This forgetting issue becomes even more severe in online task-free continual learning, where the model has to adapt to the latest information autonomously without knowing task boundaries in advance. Furthermore, the inner sparse distributed memory used to store data points can become overburdened due to the rapid growth of new data points that need to be stored for upcoming tasks. To deal with these challenges, in this article, we propose an efficient model for online task-free continual learning built on the following two novel concepts. Firstly, we introduce a new powerful operator based on Global Centroid Features (GCF) to take advantage of the condensed knowledge of data points stored in dynamic content-addressable memory. Thanks to a suitable clustering mechanism, the GCF accumulation is able to control the storage overload issue caused by the rapid increase of newly stored data points in the upcoming tasks. Secondly, we design a continual learning model that effectively leverages the extraction of GCF-based information, enabling it to alleviate catastrophic forgetting through cognitive condensation of GCF. Thereby, the proposed GCF-based model simultaneously exploits the global centroid information of the stored data points and efficiently manages the rapid growth of stored data points in the storage space. The experimental results on benchmark image classification datasets emphasize the superior performance of our approach compared to state-of-the-art methods. The code implementation of the proposed GCF-based model is publicly available at <a href="https://github.com/tutc/GCF">https://github.com/tutc/GCF</a>.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Dynamic content-addressable memory based on global centroid features for online task-free continual learning

  • Cong Tu Tran,
  • Thanh Tuan Nguyen,
  • Thanh Phuong Nguyen,
  • Nadège Thirion-Moreau

摘要

In online continual learning, a neural network learns from a continuous data stream in which each data point is processed only once and never revisited in the next learning stages. This non-revisiting mechanism often leads to Catastrophic Interference (CI), i.e., the model can dramatically lose the learned features of previous tasks when new information is learned. This forgetting issue becomes even more severe in online task-free continual learning, where the model has to adapt to the latest information autonomously without knowing task boundaries in advance. Furthermore, the inner sparse distributed memory used to store data points can become overburdened due to the rapid growth of new data points that need to be stored for upcoming tasks. To deal with these challenges, in this article, we propose an efficient model for online task-free continual learning built on the following two novel concepts. Firstly, we introduce a new powerful operator based on Global Centroid Features (GCF) to take advantage of the condensed knowledge of data points stored in dynamic content-addressable memory. Thanks to a suitable clustering mechanism, the GCF accumulation is able to control the storage overload issue caused by the rapid increase of newly stored data points in the upcoming tasks. Secondly, we design a continual learning model that effectively leverages the extraction of GCF-based information, enabling it to alleviate catastrophic forgetting through cognitive condensation of GCF. Thereby, the proposed GCF-based model simultaneously exploits the global centroid information of the stored data points and efficiently manages the rapid growth of stored data points in the storage space. The experimental results on benchmark image classification datasets emphasize the superior performance of our approach compared to state-of-the-art methods. The code implementation of the proposed GCF-based model is publicly available at https://github.com/tutc/GCF.