Neural Network Compression and Knowledge Distillation
摘要
Running large neural networksNeural network on phones, IoT devices, and embedded systemsEmbedembedded system is difficult because of limited memoryMemory, storage, compute, and battery. Neural networkNeural network compressionCompression methods aim to shrink models while keeping accuracy acceptable. This chapter surveys key methods for compressing neural networksNeural network, including pruningPruning, weight sharingWeightweight sharing, matrixMatrix decompositionDecompositionmatrix decomposition, and quantizationQuantization, with a particular focus onKnowledgeknowledge distillation knowledge distillationDistillation as a powerful approach for achieving efficient model compressionCompression. By presenting foundational concepts and recent advances, this chapter aims to provide a comprehensive understanding of how neural networksNeural network can be effectively compressed for deployment in resource-limited environments.