Triangle Regularization for Pruning Aware Training of Neural Networks
摘要
Neural networks have increasingly been adopted in practical applications, leading to a growing need for improving their computational efficiency and energy consumption. Pruning has emerged as a key technique in addressing this challenge. Traditional pruning methods typically involve training a dense neural network to completion, followed by the removal of less significant weights or neurons. However, this approach often results in suboptimal performance due to a mismatch between the training and inference phases. In this paper, we propose a pruning-aware training method based on regularization using the tent (triangular) function. Our method enables the neural network to jointly optimize for both sparsity and performance during training. We evaluate our approach on benchmark datasets, including MNIST, CIFAR-10, and CIFAR-100, demonstrating its effectiveness and superior performance compared to traditional pruning techniques.