Scholarlink

<p>Offline reinforcement learning (ORL), with the aim of learning optimal policies from offline datasets, is an important paradigm of RL. Recently, Diffusion-QL has significantly boosted the performance of offline RL using a conditional diffusion model to represent RL policy. However, experimental results showed that normalized Q-value in the Diffusion-QL significantly reduced offline RL’s performance. To solve this problem, we proposed a Diffusion Policy with Q-Ensemble (DPQE) in this study. The DPQE employs the Q-ensemble method to mitigate the overestimation of Q-values on Out-Of-Distribution actions. It uses the variance of Q-ensemble to adaptively adjust the weight of the Q-value to balance behavior cloning and Q-learning term. In addition, an ensemble gradient diversification regularization term is introduced to the naive RL loss to improve the stability of the training process. Our approach is evaluated on the D4RL Gym-MuJoCo benchmarks and experimental results outperform the State-Of-The-Art offline RL algorithms.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Diffusion policy with Q-ensemble for offline reinforcement learning

  • 期刊论文

  • 发表日期：

    2025年8月31日

查看全文
  • Dong Wang,
  • Xu Zhang,
  • Qiang Hua,
  • Chunru Dong,
  • Feng Zhang

摘要

Offline reinforcement learning (ORL), with the aim of learning optimal policies from offline datasets, is an important paradigm of RL. Recently, Diffusion-QL has significantly boosted the performance of offline RL using a conditional diffusion model to represent RL policy. However, experimental results showed that normalized Q-value in the Diffusion-QL significantly reduced offline RL’s performance. To solve this problem, we proposed a Diffusion Policy with Q-Ensemble (DPQE) in this study. The DPQE employs the Q-ensemble method to mitigate the overestimation of Q-values on Out-Of-Distribution actions. It uses the variance of Q-ensemble to adaptively adjust the weight of the Q-value to balance behavior cloning and Q-learning term. In addition, an ensemble gradient diversification regularization term is introduced to the naive RL loss to improve the stability of the training process. Our approach is evaluated on the D4RL Gym-MuJoCo benchmarks and experimental results outperform the State-Of-The-Art offline RL algorithms.