<p>The pi-sigma network (PSN), as a high-order network, has demonstrated its capacity for rapid learning and strong nonlinear processing. This paper proposes a type algorithm, for PSNs using a batch gradient method based on <InlineEquation ID="IEq1"> <EquationSource Format="TEX">\({\text{L}}_{1}\)</EquationSource> </InlineEquation> regularization. Direct application of <InlineEquation ID="IEq2"> <EquationSource Format="TEX">\({\text{L}}_{1}\)</EquationSource> </InlineEquation> regularization during network training presents two main drawbacks. There are numerical oscillations and theoretical challenges in computing the gradients at the origin. We then introduced smoothing functions by approximating the <InlineEquation ID="IEq3"> <EquationSource Format="TEX">\({\text{L}}_{1}\)</EquationSource> </InlineEquation> regularization to overcome these obstacles, resulting in a new gradient descent method based on smoothing <InlineEquation ID="IEq4"> <EquationSource Format="TEX">\({\text{L}}_{1}\)</EquationSource> </InlineEquation> regularization (GDS<InlineEquation ID="IEq5"> <EquationSource Format="TEX">\({\text{L}}_{1}\)</EquationSource> </InlineEquation>). Numerical results for the 4-dimensional parity problem and the nonlinear Gabor function problem demonstrate that the GDS<InlineEquation ID="IEq6"> <EquationSource Format="TEX">\({\text{L}}_{1}\)</EquationSource> </InlineEquation> algorithms perform better than the other four regularization methods in terms of generalization and pruning efficiency. Theoretical analysis and experimental verification strictly prove the monotonicity and strong and weak convergence of the network based on the GDS<InlineEquation ID="IEq7"> <EquationSource Format="TEX">\({\text{L}}_{1}\)</EquationSource> </InlineEquation> algorithms.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Convergence analysis and application for high-order neural networks based on gradient descent learning algorithm via smooth regularization

  • Khidir Shaib Mohamed,
  • Alawia Adam,
  • Yousif Shoaib Mohammed,
  • Yan Xiong

摘要

The pi-sigma network (PSN), as a high-order network, has demonstrated its capacity for rapid learning and strong nonlinear processing. This paper proposes a type algorithm, for PSNs using a batch gradient method based on \({\text{L}}_{1}\) regularization. Direct application of \({\text{L}}_{1}\) regularization during network training presents two main drawbacks. There are numerical oscillations and theoretical challenges in computing the gradients at the origin. We then introduced smoothing functions by approximating the \({\text{L}}_{1}\) regularization to overcome these obstacles, resulting in a new gradient descent method based on smoothing \({\text{L}}_{1}\) regularization (GDS \({\text{L}}_{1}\) ). Numerical results for the 4-dimensional parity problem and the nonlinear Gabor function problem demonstrate that the GDS \({\text{L}}_{1}\) algorithms perform better than the other four regularization methods in terms of generalization and pruning efficiency. Theoretical analysis and experimental verification strictly prove the monotonicity and strong and weak convergence of the network based on the GDS \({\text{L}}_{1}\) algorithms.