The aim of this paper is to analyze a different practical implementation of the LIne Search based stochastic gradient Algorithm (LISA) recently proposed by Franchini et al. The LISA scheme belongs to the class of stochastic gradient methods and it practically relies on a line search strategy to select the learning rate and a dynamic technique to increase the mini batch size of the stochastic directions. Despite promising performance of LISA in solving optimization problems from machine learning applications, its mini batch increasing strategy involves checking for a condition which may be computationally expensive and memory demanding, especially in the presence of both deep neural networks and very large-scale dataset. In this work we investigate an a-priori procedure to select the size of the current mini batch which allows to dampen the computational execution time of the LISA method and to control the requests of memory resources, especially when dealing with hardware accelerators. A numerical experience on training both statistical models for binary classification and deep neural networks for multi-class image classification confirms the effectiveness of the proposal.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Line Search Stochastic Gradient Algorithm with A-priori Rule for Monitoring the Control of the Variance

  • Giorgia Franchini,
  • Federica Porta,
  • Valeria Ruggiero,
  • Ilaria Trombini,
  • Luca Zanni

摘要

The aim of this paper is to analyze a different practical implementation of the LIne Search based stochastic gradient Algorithm (LISA) recently proposed by Franchini et al. The LISA scheme belongs to the class of stochastic gradient methods and it practically relies on a line search strategy to select the learning rate and a dynamic technique to increase the mini batch size of the stochastic directions. Despite promising performance of LISA in solving optimization problems from machine learning applications, its mini batch increasing strategy involves checking for a condition which may be computationally expensive and memory demanding, especially in the presence of both deep neural networks and very large-scale dataset. In this work we investigate an a-priori procedure to select the size of the current mini batch which allows to dampen the computational execution time of the LISA method and to control the requests of memory resources, especially when dealing with hardware accelerators. A numerical experience on training both statistical models for binary classification and deep neural networks for multi-class image classification confirms the effectiveness of the proposal.