Scholarlink

This chapter explores prominent theoretical frameworks used to understand learning and generalizationGeneralization in deepLearningdeep learning learningDeepdeep learning. It covers Probably Approximately Correct (PAC) learning, errorErrorerror bound boundsBounderror bound and generalizationGeneralization in neural networksNeural network, theInformationinformation bottleneck information bottleneckBottleneck principle, double descentDouble descent, grokkingGrokking, benign overfittingBenign overfitting, implicit biasBias ofGradientstochastic gradient descent stochastic gradient Stochasticstochastic gradient descent descentDescentstochastic gradient descent, neural networkNeural network kernel theory, lottery ticket hypothesisLottery ticket hypothesis, and scaling lawsScaling laws. First, it introduces PAC learning LearningPAC learning theory as a statisticalStatistical framework that aims to bound the true (test)Test error in terms of the empiricalEmpirical (training) error. Concepts such as realizable and agnostic PAC learnabilityProbably Approximately Correct (PAC) learning/learnabilityagnostic PAC learnability, VC dimensionVC dimension, and complexityComplexity-dependent Errorerror bound error boundsBounderror bound are discussed in the context of their applicability to neural networksNeural network. However, the chapter emphasizes a fundamental disconnect between PAC learningLearningPAC learning theory and the empiricalEmpirical success of modern over-parameterizedOver-parameterized deep networks, demonstrating that PAC theory does not fully account for their generalizationGeneralization behavior. The second part presents the information-theoretic perspective, wherein a neural networkNeural network is viewed as anInformationinformation bottleneck information bottleneckBottleneck that compresses input data while preserving only the most taskTask-relevant information. This viewpoint offers an alternative explanation for generalizationGeneralization in deepLearningdeep learning learningDeepdeep learning, especially in highly over-parameterizedOver-parameterized settings. Collectively, these two theoretical approaches provide complementary insights into the challenges and open questions surrounding the effectiveness of deepLearningdeep learning learningDeepdeep learning. Finally, the third part of chapter briefly introduces a few theoretical advances in deep Learningdeep learning learning Deepdeep learning generalizationGeneralization, including double descentDouble descent, grokkingGrokking, benign overfittingBenign overfitting, implicit biasBias ofGradientstochastic gradient descent stochastic gradient Stochasticstochastic gradient descent descentDescentstochastic gradient descent, neural networkNeural network kernel theory, lottery ticket hypothesisLottery ticket hypothesis, and scaling lawsScaling laws. These concepts explain why deep neural networksDeepdeep neural network often exhibit good generalizationGeneralization to unseen data.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Theoretical Foundations of Generalization in Deep Learning  A chemical structure showing a molecule with a central six-membered ring containing alternating double bonds and nitrogen atoms at two positions. Attached to this ring are various groups, including a benzene ring, a carbonyl group, and a side chain with nitrogen and oxygen atoms. The structure highlights the arrangement of atoms and bonds important for understanding the molecule’s chemical properties and potential biological activity.

  • 章节

  • 发表日期：

    2026年

查看全文
  • Benyamin Ghojogh,
  • Ali Ghodsi

摘要

This chapter explores prominent theoretical frameworks used to understand learning and generalizationGeneralization in deepLearningdeep learning learningDeepdeep learning. It covers Probably Approximately Correct (PAC) learning, errorErrorerror bound boundsBounderror bound and generalizationGeneralization in neural networksNeural network, theInformationinformation bottleneck information bottleneckBottleneck principle, double descentDouble descent, grokkingGrokking, benign overfittingBenign overfitting, implicit biasBias ofGradientstochastic gradient descent stochastic gradient Stochasticstochastic gradient descent descentDescentstochastic gradient descent, neural networkNeural network kernel theory, lottery ticket hypothesisLottery ticket hypothesis, and scaling lawsScaling laws. First, it introduces PAC learning LearningPAC learning theory as a statisticalStatistical framework that aims to bound the true (test)Test error in terms of the empiricalEmpirical (training) error. Concepts such as realizable and agnostic PAC learnabilityProbably Approximately Correct (PAC) learning/learnabilityagnostic PAC learnability, VC dimensionVC dimension, and complexityComplexity-dependent Errorerror bound error boundsBounderror bound are discussed in the context of their applicability to neural networksNeural network. However, the chapter emphasizes a fundamental disconnect between PAC learningLearningPAC learning theory and the empiricalEmpirical success of modern over-parameterizedOver-parameterized deep networks, demonstrating that PAC theory does not fully account for their generalizationGeneralization behavior. The second part presents the information-theoretic perspective, wherein a neural networkNeural network is viewed as anInformationinformation bottleneck information bottleneckBottleneck that compresses input data while preserving only the most taskTask-relevant information. This viewpoint offers an alternative explanation for generalizationGeneralization in deepLearningdeep learning learningDeepdeep learning, especially in highly over-parameterizedOver-parameterized settings. Collectively, these two theoretical approaches provide complementary insights into the challenges and open questions surrounding the effectiveness of deepLearningdeep learning learningDeepdeep learning. Finally, the third part of chapter briefly introduces a few theoretical advances in deep Learningdeep learning learning Deepdeep learning generalizationGeneralization, including double descentDouble descent, grokkingGrokking, benign overfittingBenign overfitting, implicit biasBias ofGradientstochastic gradient descent stochastic gradient Stochasticstochastic gradient descent descentDescentstochastic gradient descent, neural networkNeural network kernel theory, lottery ticket hypothesisLottery ticket hypothesis, and scaling lawsScaling laws. These concepts explain why deep neural networksDeepdeep neural network often exhibit good generalizationGeneralization to unseen data.