Scholarlink

<p>Exploration has long been a challenge in reinforcement learning, particularly in sparse reward environments. The entropy-based exploration strategies have been proved to be a kind of promising methods which promote uniform coverage of the state space by maximizing the entropy of the state distribution (i.e., state entropy). However, most existing entropy-based exploration suffer from the issue of inaccurate intrinsic reward estimation and reward vanishing. To address these issues, we propose a novel intrinsic reward mechanism termed Adaptive Joint Entropy Reward (AJER), which features three key technical contributions:(1) A dedicated representation learning module specifically designed for accurate intrinsic reward estimation; (2) A novel fused entropy reward that, for the first time, combines Rényi entropy (excelling at incentivizing exploration of hard-to-reach states) and Shannon entropy (providing stability and uniform coverage) to mitigate reward vanishing and enhance exploration drive; (3) A fully adaptive fusion mechanism that automatically balances the contributions of the two entropy sources without requiring manual weight tuning for different tasks. The design of AJER is grounded in the principle of generalized maximum entropy exploration, providing a theoretical foundation for its improved performance. Enabled by the more accurate feature extraction of its representation learning module and the stronger exploration drive of Rényi entropy, AJER can effectively guide agents to make better decisions in sparse reward environments. We evaluated our method on the MiniGrid and the DeepMind Control Suite benchmarks, covering a wide range of tasks, such as discrete and continuous, simple and complex environments. Experimental results demonstrate that the AJER significantly improved performance in various tasks. For example, compared with the current state-of-the-art methods (i.e., SE and VCSE), our method achieved improvements of 21.46%, 47.7%, and 11.4% in DoorKey-6x6, Quadruped Walk, and Walker Run tasks, respectively, though at the cost of increased computational overhead from its representation learning module and dual entropy calculations. The code is available at the following link: <a href="https://github.com/wyang-up/JER">https://github.com/wyang-up/JER</a>.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Adaptive joint entropy reward: a mechanism to efficient exploration in reinforcement learning

  • 期刊论文

  • 发表日期：

    2025年11月19日

查看全文
  • Chunru Dong,
  • Yang Wang,
  • Aoxiang Wang,
  • Qiang Hua,
  • Feng Zhang

摘要

Exploration has long been a challenge in reinforcement learning, particularly in sparse reward environments. The entropy-based exploration strategies have been proved to be a kind of promising methods which promote uniform coverage of the state space by maximizing the entropy of the state distribution (i.e., state entropy). However, most existing entropy-based exploration suffer from the issue of inaccurate intrinsic reward estimation and reward vanishing. To address these issues, we propose a novel intrinsic reward mechanism termed Adaptive Joint Entropy Reward (AJER), which features three key technical contributions:(1) A dedicated representation learning module specifically designed for accurate intrinsic reward estimation; (2) A novel fused entropy reward that, for the first time, combines Rényi entropy (excelling at incentivizing exploration of hard-to-reach states) and Shannon entropy (providing stability and uniform coverage) to mitigate reward vanishing and enhance exploration drive; (3) A fully adaptive fusion mechanism that automatically balances the contributions of the two entropy sources without requiring manual weight tuning for different tasks. The design of AJER is grounded in the principle of generalized maximum entropy exploration, providing a theoretical foundation for its improved performance. Enabled by the more accurate feature extraction of its representation learning module and the stronger exploration drive of Rényi entropy, AJER can effectively guide agents to make better decisions in sparse reward environments. We evaluated our method on the MiniGrid and the DeepMind Control Suite benchmarks, covering a wide range of tasks, such as discrete and continuous, simple and complex environments. Experimental results demonstrate that the AJER significantly improved performance in various tasks. For example, compared with the current state-of-the-art methods (i.e., SE and VCSE), our method achieved improvements of 21.46%, 47.7%, and 11.4% in DoorKey-6x6, Quadruped Walk, and Walker Run tasks, respectively, though at the cost of increased computational overhead from its representation learning module and dual entropy calculations. The code is available at the following link: https://github.com/wyang-up/JER.