Parameter reduction in convolutional neural networks with kernel transposition
摘要
Deep convolutional neural networks (CNNs) are a popular choice for many image classification tasks due to their good performance. In general, there is a correlation between the generalization performance of a deep learning model, the number of training samples available, and the number of free parameters in the model. If there are too many free parameters in the model relative to the amount of training data, the model overfits the training data. Reducing the number of trainable parameters in CNNs while maintaining performance levels is an active area of research. This is important, especially in resource-constrained environments where either the size of the training data is insufficient or the memory of the hardware where the model is to be deployed is limited, such as in mobile vision applications. In this paper, kernel transposition is proposed as a method for the reduction of free parameters in the model’s convolutional layers. This allows kernel reuse, in which a model learns a given kernel only once but uses it twice. The learned kernels and their transposes can either be used in sequence or in parallel to one another. These form series and parallel convolutional modules respectively. The modules are used as replacements of the traditional convolutional layers in a CNN model. The use of the modules reduces the number of free parameters and computational costs associated with convolution operations in the model without necessarily compromising its generalization performance. The proposed method is generic and can be adapted to existing state-of-the-art network architectures as demonstrated in the experiments using established architectures. The proposed method was validated on the CIFAR-10 and CIFAR-100 standard datasets using a series of experiments with five model architectures: a basic 5-convolutional layer CNN model, the ResNet-56, small MobileNetV3, EfficientNetB0 and ConvNext model architectures; all trained from scratch. Models based on standard convolutional layers (for basic CNN and ResNet-56 architectures) and depth-wise convolutional layers (for MobileNetV3, EfficientNetB0 and ConvNext architectures) were compared against those based on the proposed convolutional modules. The models based on the standard and depth-wise convolutional layers were used as baselines. The comparisons were in terms of the number of parameters, computational efficiencies (Floating Point Operations / FLOPs) and classification accuracies of the models. For a kernel size k, the experiments showed that, despite improvements on parameter and computational efficiencies, models based on the proposed modules for the basic CNN (k = 3) architecture had lower accuracy levels compared to their baseline model. However, the huge saving on the number of parameters (> 88%) and FLOPs (> 84%) is significant compared to the drop in accuracy (< 4%). For the MobileNetV3 (k = 3 or k = 5) and EfficientNetB0 (k = 3 or k = 5), only the model based on the series module had lower accuracy than the baseline models. The models based on the parallel module were generally at par (within a margin of