Scholarlink

<p>One of the central challenges in modern machine learning is understanding how neural networks generalize knowledge learned from training data to unseen test data. While numerous empirical techniques have been proposed to improve generalization, a theoretical understanding of its mechanism remains elusive. Here we introduce the concept of Boltzmann entropy into neural networks. By employing molecular simulation algorithms, we compute entropy landscapes as functions of both training loss and test accuracy (or test loss) across four distinct machine learning tasks. Our results reveal the existence of <i>high-entropy advantage</i>, wherein high-entropy network states generally outperform those reached via conventional training techniques like stochastic gradient descent. This entropy advantage provides a thermodynamic explanation for neural network generalizability: the generalizable states occupy a larger part of the parameter space than its non-generalizable analog at low train loss. We also find this advantage more pronounced in narrower neural networks.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

High-entropy advantage in neural networks' generalizability

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年4月16日

查看全文
  • Entao Yang,
  • Xiaotian Zhang,
  • Yue Shang,
  • Ge Zhang

摘要

One of the central challenges in modern machine learning is understanding how neural networks generalize knowledge learned from training data to unseen test data. While numerous empirical techniques have been proposed to improve generalization, a theoretical understanding of its mechanism remains elusive. Here we introduce the concept of Boltzmann entropy into neural networks. By employing molecular simulation algorithms, we compute entropy landscapes as functions of both training loss and test accuracy (or test loss) across four distinct machine learning tasks. Our results reveal the existence of high-entropy advantage, wherein high-entropy network states generally outperform those reached via conventional training techniques like stochastic gradient descent. This entropy advantage provides a thermodynamic explanation for neural network generalizability: the generalizable states occupy a larger part of the parameter space than its non-generalizable analog at low train loss. We also find this advantage more pronounced in narrower neural networks.