Optimization of finite-sum problems has become fundamental in many fields, particularly in machine learning and data science. These problems, which are often of the form \(\displaystyle \begin{aligned} {} \min _{x\in {\mathbb R}^d} \left \{f(x) = \frac {1}{N} \sum _{i=1}^N f_i(x)\right \}, \end{aligned} \) involve minimizing the average of a large number of component functions \(f_i(x)\) , each typically representing the loss associated with a single data point. Such problems arise frequently in empirical risk minimization, where \(f_i(x)\) is the loss of a predictive model on the i-th training example. When the number of components N is large, computing the full gradient \(\nabla f(x)\) at each iteration of traditional gradient-based methods becomes computationally expensive, motivating the use of stochastic algorithms that offer more efficient alternatives.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Stochastic Algorithms

  • Qinian Jin

摘要

Optimization of finite-sum problems has become fundamental in many fields, particularly in machine learning and data science. These problems, which are often of the form \(\displaystyle \begin{aligned} {} \min _{x\in {\mathbb R}^d} \left \{f(x) = \frac {1}{N} \sum _{i=1}^N f_i(x)\right \}, \end{aligned} \) involve minimizing the average of a large number of component functions \(f_i(x)\) , each typically representing the loss associated with a single data point. Such problems arise frequently in empirical risk minimization, where \(f_i(x)\) is the loss of a predictive model on the i-th training example. When the number of components N is large, computing the full gradient \(\nabla f(x)\) at each iteration of traditional gradient-based methods becomes computationally expensive, motivating the use of stochastic algorithms that offer more efficient alternatives.