Scholarlink

This paper addresses the issue of training instability and weak generalization faced in distributional reinforcement learning for continuous control. We augment Distributed Distributional Deep Deterministic Policy Gradient (D4PG) with an Implicit Quantile Network (IQN) critic using 3 mechanisms: adaptive quantile sampling via a temperature-controlled softmax over quantile Q-values, Elastic Weight Consolidation (EWC) whose strength adapts to recent performance, and entropy with mild action-smoothing regularization to sustain exploration. On Pendulum with 3 seeds, gains are limited where adaptive quantile sampling yields +0.33% over baseline, while EWC (−3.31%) and the full stack (−7.23%) underperform, reflecting minimal headroom in a simple task. On the LunarLanderContinuous, EWC raises mean return by +39.4%, and the full stack delivers +8.0% with smoother learning curves. A preliminary HalfCheetah trial further suggests improved sample-efficiency where the enhanced agent reaches a last-100-episode average of 9757.53 vs 6946.9 for baseline after 1007 vs 2000 episodes, with an overall mean 6317.79 vs 4825.56 (+30.92%). These results indicate that adaptively focusing the return distribution and protecting critical parameters improves stability and robustness as task complexity increases, while entropy trades a small amount of asymptotic return for broader exploration and policy resilience.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Enhancing Reinforcement Learning Performance and Stability through Adaptive Quantile Sampling, Elastic Weight Consolidation, and Multi-objective Regularization

  • 会议论文

  • 发表日期：

    2026年

查看全文
  • Kai Ping Hoo,
  • Tomas Maul

摘要

This paper addresses the issue of training instability and weak generalization faced in distributional reinforcement learning for continuous control. We augment Distributed Distributional Deep Deterministic Policy Gradient (D4PG) with an Implicit Quantile Network (IQN) critic using 3 mechanisms: adaptive quantile sampling via a temperature-controlled softmax over quantile Q-values, Elastic Weight Consolidation (EWC) whose strength adapts to recent performance, and entropy with mild action-smoothing regularization to sustain exploration. On Pendulum with 3 seeds, gains are limited where adaptive quantile sampling yields +0.33% over baseline, while EWC (−3.31%) and the full stack (−7.23%) underperform, reflecting minimal headroom in a simple task. On the LunarLanderContinuous, EWC raises mean return by +39.4%, and the full stack delivers +8.0% with smoother learning curves. A preliminary HalfCheetah trial further suggests improved sample-efficiency where the enhanced agent reaches a last-100-episode average of 9757.53 vs 6946.9 for baseline after 1007 vs 2000 episodes, with an overall mean 6317.79 vs 4825.56 (+30.92%). These results indicate that adaptively focusing the return distribution and protecting critical parameters improves stability and robustness as task complexity increases, while entropy trades a small amount of asymptotic return for broader exploration and policy resilience.