Stepsize plays an important role in the stochastic gradient method. The bandwidth-based stepsize allows us to adjust the stepsize within a banded region determined by some boundary functions. Based on the bandwidth-based stepsize, we propose a new method, namely SCSG-BD, for smooth non-convex finite-sum optimization problems. For the boundary functions 1/t, <InlineEquation ID="IEq1"> <InlineMediaObject> <ImageObject Color="BlackWhite" FileRef="10589_2025_651_Article_IEq1.gif" Format="GIF" Height="19" Rendition="HTML" Resolution="72" Type="Linedraw" Width="106" /> </InlineMediaObject> <EquationSource Format="TEX">$1/(t\log (t + 1))$</EquationSource> <EquationSource Format="MATHML"><math> <mrow> <mn>1</mn> <mo stretchy="false">/</mo> <mo stretchy="false">(</mo> <mi>t</mi> <mo>log</mo> <mo stretchy="false">(</mo> <mi>t</mi> <mo>+</mo> <mn>1</mn> <mo stretchy="false">)</mo> <mo stretchy="false">)</mo> </mrow> </math></EquationSource> </InlineEquation> and <InlineEquation ID="IEq2"> <InlineMediaObject> <ImageObject Color="BlackWhite" FileRef="10589_2025_651_Article_IEq2.gif" Format="GIF" Height="19" Rendition="HTML" Resolution="72" Type="Linedraw" Width="31" /> </InlineMediaObject> <EquationSource Format="TEX">$1/t^p$</EquationSource> <EquationSource Format="MATHML"><math> <mrow> <mn>1</mn> <mo stretchy="false">/</mo> <msup> <mi>t</mi> <mi>p</mi> </msup> </mrow> </math></EquationSource> </InlineEquation> (<InlineEquation ID="IEq3"> <InlineMediaObject> <ImageObject Color="BlackWhite" FileRef="10589_2025_651_Article_IEq3.gif" Format="GIF" Height="19" Rendition="HTML" Resolution="72" Type="Linedraw" Width="68" /> </InlineMediaObject> <EquationSource Format="TEX">$p\in (0,1)$</EquationSource> <EquationSource Format="MATHML"><math> <mrow> <mi>p</mi> <mo>∈</mo> <mo stretchy="false">(</mo> <mn>0</mn> <mo>,</mo> <mn>1</mn> <mo stretchy="false">)</mo> </mrow> </math></EquationSource> </InlineEquation>), SCSG-BD converges sublinearly to a stationary point at a faster rate than the stochastically controlled stochastic gradient (SCSG) method under certain conditions. Moreover, SCSG-BD is able to converge linearly to the solution if the objective function satisfies the Polyak–Łojasiewicz condition. We also introduce the 1/t-Barzilai–Borwein stepsize for practical computation. Numerical experiments demonstrate that SCSG-BD performs better than SCSG and its variants.

Improving the stochastically controlled stochastic gradient method by the bandwidth-based stepsize

期刊论文
发表日期：

2025年1月23日

查看全文

Chenchen Liu,
Yakui Huang,
Dan Wang

摘要

Stepsize plays an important role in the stochastic gradient method. The bandwidth-based stepsize allows us to adjust the stepsize within a banded region determined by some boundary functions. Based on the bandwidth-based stepsize, we propose a new method, namely SCSG-BD, for smooth non-convex finite-sum optimization problems. For the boundary functions 1/t, $1/(t\log (t + 1))$ $1 / (t log (t + 1))$ and $1/t^p$ $1 / t^{p}$ ( $p\in (0,1)$ $p \in (0, 1)$ ), SCSG-BD converges sublinearly to a stationary point at a faster rate than the stochastically controlled stochastic gradient (SCSG) method under certain conditions. Moreover, SCSG-BD is able to converge linearly to the solution if the objective function satisfies the Polyak–Łojasiewicz condition. We also introduce the 1/t-Barzilai–Borwein stepsize for practical computation. Numerical experiments demonstrate that SCSG-BD performs better than SCSG and its variants.