Scholarlink

Abstract <p>Deep learning networks have been trained using first-order-based methods. These methods often converge more quickly when combined with an adaptive step size, but they tend to settle at suboptimal points, especially when learning occurs in a large output space. When first-order-based methods are used with a constant step size, they oscillate near the zero-gradient region, which leads to slow convergence. However, these issues are exacerbated under nonconvexity, which can significantly diminish the performance of first-order methods. In this work, we propose a novel&#xa0;<b>B</b>oltzmann Probability Weighted&#xa0;<b>S</b>ine with a&#xa0;<b>C</b>osine distance-based&#xa0;<b>A</b>daptive&#xa0;<b>Grad</b>ient (<i>BSCAGrad</i>) method. The step size in this method is carefully designed to mitigate the issue of slow convergence. Furthermore, it facilitates escape from suboptimal points, enabling the optimization process to progress more efficiently toward local minima. This is achieved by combining a Boltzmann probability-weighted sine function and cosine distance to calculate the step size. The Boltzmann probability-weighted sine function acts when the gradient vanishes and the cooling parameter remains moderate, a condition typically observed near suboptimal points. Moreover, using the sine function on the exponential moving average of the weight parameters leverages geometric information from the data. The cosine distance prevents zero in the step size. Together, these components accelerate convergence, improve stability, and guide the algorithm toward a better optimal solution. A theoretical analysis of the convergence rate under both convexity and nonconvexity is provided to substantiate the findings. The experimental results from language modeling, object detection, machine translation, and image classification tasks on a real-world benchmark dataset, including <i>CIFAR</i>10, <i>CIFAR</i>100, <i>PennTreeBank</i>, <i>PASCALVOC</i> and <i>WMT</i>2014, demonstrate that the proposed step size outperforms traditional baseline methods.</p> Graphical abstract <p></p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Sine and cosine based learning rate for gradient descent method

  • 期刊论文

  • 发表日期：

    2025年1月20日

查看全文
  • Krutika Verma,
  • Abyayananda Maiti

摘要

Abstract

Deep learning networks have been trained using first-order-based methods. These methods often converge more quickly when combined with an adaptive step size, but they tend to settle at suboptimal points, especially when learning occurs in a large output space. When first-order-based methods are used with a constant step size, they oscillate near the zero-gradient region, which leads to slow convergence. However, these issues are exacerbated under nonconvexity, which can significantly diminish the performance of first-order methods. In this work, we propose a novel Boltzmann Probability Weighted Sine with a Cosine distance-based Adaptive Gradient (BSCAGrad) method. The step size in this method is carefully designed to mitigate the issue of slow convergence. Furthermore, it facilitates escape from suboptimal points, enabling the optimization process to progress more efficiently toward local minima. This is achieved by combining a Boltzmann probability-weighted sine function and cosine distance to calculate the step size. The Boltzmann probability-weighted sine function acts when the gradient vanishes and the cooling parameter remains moderate, a condition typically observed near suboptimal points. Moreover, using the sine function on the exponential moving average of the weight parameters leverages geometric information from the data. The cosine distance prevents zero in the step size. Together, these components accelerate convergence, improve stability, and guide the algorithm toward a better optimal solution. A theoretical analysis of the convergence rate under both convexity and nonconvexity is provided to substantiate the findings. The experimental results from language modeling, object detection, machine translation, and image classification tasks on a real-world benchmark dataset, including CIFAR10, CIFAR100, PennTreeBank, PASCALVOC and WMT2014, demonstrate that the proposed step size outperforms traditional baseline methods.

Graphical abstract