Meta-reinforcement learning is a rapidly developing field in AI, designed to address complex application domains with similar tasks, enabling quick mastery of tasks within a specific domain through meta-learning. However, existing meta-RL algorithms generally suffer from low efficiency and are mostly applied to low-dimensional task distributions, performing poorly in high-dimensional tasks. Meanwhile, Model-Agnostic Meta-Learning (MAML) is one of the best meta methods in the meta-algorithms field. MAML is easy to deploy, widely applicable, and powerful, but it also has several issues, such as being highly sensitive to neural network architectures during training, which often leads to instability, requiring tedious hyperparameter searches to stabilize training, and incurring high computational costs during both training and inference. As the number of subtasks increases, MAML becomes difficult to generalize effectively for meta-learning, and the model tends to memorize periodic data rather than deeply understanding the underlying principles behind the periodicity. This paper proposes FMAN (Fourier Model-Agnostic Network), which aims to improve the performance and sample efficiency of meta-learning by using Fourier Transform (FT) to extract frequency-domain information from continuous space. Specifically, I extract frequency-domain information from time-series data and process it using the actor-critic algorithm to help the agent make better decisions. One of the appealing features of this network is that it can learn underlying periodic patterns without the need to predict infinitely long states. Experiments show that FMAN offers a promising solution for improving sample efficiency and generalization in meta-worlds, establishing it as a standard benchmark in meta-learning.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Fourier Model Agnostic Meta-Reinforcement Learning Network

  • Zhe Hao,
  • Zhiheng Lai,
  • Xuanyu Xue

摘要

Meta-reinforcement learning is a rapidly developing field in AI, designed to address complex application domains with similar tasks, enabling quick mastery of tasks within a specific domain through meta-learning. However, existing meta-RL algorithms generally suffer from low efficiency and are mostly applied to low-dimensional task distributions, performing poorly in high-dimensional tasks. Meanwhile, Model-Agnostic Meta-Learning (MAML) is one of the best meta methods in the meta-algorithms field. MAML is easy to deploy, widely applicable, and powerful, but it also has several issues, such as being highly sensitive to neural network architectures during training, which often leads to instability, requiring tedious hyperparameter searches to stabilize training, and incurring high computational costs during both training and inference. As the number of subtasks increases, MAML becomes difficult to generalize effectively for meta-learning, and the model tends to memorize periodic data rather than deeply understanding the underlying principles behind the periodicity. This paper proposes FMAN (Fourier Model-Agnostic Network), which aims to improve the performance and sample efficiency of meta-learning by using Fourier Transform (FT) to extract frequency-domain information from continuous space. Specifically, I extract frequency-domain information from time-series data and process it using the actor-critic algorithm to help the agent make better decisions. One of the appealing features of this network is that it can learn underlying periodic patterns without the need to predict infinitely long states. Experiments show that FMAN offers a promising solution for improving sample efficiency and generalization in meta-worlds, establishing it as a standard benchmark in meta-learning.