<p>Recent advances have introduced inertial extrapolation steps to enhance algorithm convergence speed. Inspired by the remarkable efficacy of stochastic conjugate gradient (SCG) methods, this study introduces a novel biased stochastic conjugate gradient algorithm with inertial extrapolation to address machine learning challenges. The integration of the Modified Polak-Ribiére-Polyak conjugate gradient (MPRP CG) algorithm enables this method to overcome the traditional descending direction assumption by incorporating the trust region concept. Extrapolation steps in both step size and direction endow the algorithm with a flexible acceleration mechanism, promising substantial enhancements in convergence speed. This paper presents theoretical proof of the algorithm’s convergence to the local optimum and establishes a framework for its linear convergence rate with non-convex objectives. Numerical experiments on machine learning models extensively validate the superior performance of the introduced algorithm.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Stochastic conjugate gradient algorithm with an inertial extrapolation step for nonconvex optimization in machine learning

  • Yijia Wang,
  • Chen Ouyang,
  • Beisai Hu,
  • Gonglin Yuan,
  • Yunfeng Wang

摘要

Recent advances have introduced inertial extrapolation steps to enhance algorithm convergence speed. Inspired by the remarkable efficacy of stochastic conjugate gradient (SCG) methods, this study introduces a novel biased stochastic conjugate gradient algorithm with inertial extrapolation to address machine learning challenges. The integration of the Modified Polak-Ribiére-Polyak conjugate gradient (MPRP CG) algorithm enables this method to overcome the traditional descending direction assumption by incorporating the trust region concept. Extrapolation steps in both step size and direction endow the algorithm with a flexible acceleration mechanism, promising substantial enhancements in convergence speed. This paper presents theoretical proof of the algorithm’s convergence to the local optimum and establishes a framework for its linear convergence rate with non-convex objectives. Numerical experiments on machine learning models extensively validate the superior performance of the introduced algorithm.