Stochastic conjugate gradient algorithm with an inertial extrapolation step for nonconvex optimization in machine learning
摘要
Recent advances have introduced inertial extrapolation steps to enhance algorithm convergence speed. Inspired by the remarkable efficacy of stochastic conjugate gradient (SCG) methods, this study introduces a novel biased stochastic conjugate gradient algorithm with inertial extrapolation to address machine learning challenges. The integration of the Modified Polak-Ribiére-Polyak conjugate gradient (MPRP CG) algorithm enables this method to overcome the traditional descending direction assumption by incorporating the trust region concept. Extrapolation steps in both step size and direction endow the algorithm with a flexible acceleration mechanism, promising substantial enhancements in convergence speed. This paper presents theoretical proof of the algorithm’s convergence to the local optimum and establishes a framework for its linear convergence rate with non-convex objectives. Numerical experiments on machine learning models extensively validate the superior performance of the introduced algorithm.