Scholarlink

<p>Although deep reinforcement learning has achieved notable practical achievements, its theoretical foundations have been scarcely explored until recent times. Nonetheless, the rate of convergence for current neural temporal-difference (TD) learning algorithms is constrained, largely due to their high sensitivity to stepsize choices. In order to mitigate this issue, we propose an adaptive neural TD algorithm (<b>AdaBNTD</b>) inspired by the superior performance of adaptive gradient techniques in training deep neural networks. Simultaneously, we derive non-asymptotic bounds for <b>AdaBNTD</b> within the Markovian observation framework. In particular, <b>AdaBNTD</b> is capable of converging to the global optimum of the mean square projection Bellman error (MSPBE) with a convergence rate of <InlineEquation ID="IEq1"> <InlineMediaObject> <ImageObject Color="BlackWhite" FileRef="40747_2024_1757_Article_IEq1.gif" Format="GIF" Height="21" Rendition="HTML" Resolution="72" Type="Linedraw" Width="74" /> </InlineMediaObject> <EquationSource Format="TEX">\({{\mathcal {O}}}(1/\sqrt{K})\)</EquationSource> <EquationSource Format="MATHML"><math> <mrow> <mi mathvariant="script">O</mi> <mo stretchy="false">(</mo> <mn>1</mn> <mo stretchy="false">/</mo> <msqrt> <mi>K</mi> </msqrt> <mo stretchy="false">)</mo> </mrow> </math></EquationSource> </InlineEquation>, where <i>K</i> denotes the iteration count. Besides, the effectiveness <b>AdaBNTD</b> is also verified through several reinforcement learning benchmark domains.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Adaptive temporal-difference learning via deep neural network function approximation: a non-asymptotic analysis

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2025年1月15日

查看全文
  • Guoyong Wang,
  • Tiange Fu,
  • Ruijuan Zheng,
  • Xuhui Zhao,
  • Junlong Zhu,
  • Mingchuan Zhang

摘要

Although deep reinforcement learning has achieved notable practical achievements, its theoretical foundations have been scarcely explored until recent times. Nonetheless, the rate of convergence for current neural temporal-difference (TD) learning algorithms is constrained, largely due to their high sensitivity to stepsize choices. In order to mitigate this issue, we propose an adaptive neural TD algorithm (AdaBNTD) inspired by the superior performance of adaptive gradient techniques in training deep neural networks. Simultaneously, we derive non-asymptotic bounds for AdaBNTD within the Markovian observation framework. In particular, AdaBNTD is capable of converging to the global optimum of the mean square projection Bellman error (MSPBE) with a convergence rate of \({{\mathcal {O}}}(1/\sqrt{K})\) O ( 1 / K ) , where K denotes the iteration count. Besides, the effectiveness AdaBNTD is also verified through several reinforcement learning benchmark domains.