Deep learning (DL) has become a successful approach to solving various problems, especially within computer vision, and has applications in numerous areas. Convolutional Neural Networks (CNNs), which consist of multiple layers, are commonly used in image processing and have shown superior prediction performance compared to previous methods. Semi-Infinite Programming (SIP) is a class of problems where a restricted count of variables is subject to infinite constraints. This study proposes an optimization model based on SIP to prune an ensemble of CNNs by considering both squared loss and Shannon Entropy functions. The objective is to balance the ensemble’s accuracy and diversity, which is optimized by defining the trade-off parameter as the index of an infinite index set of constraints. To avoid the dependence on pruning size, each network in the ensemble is assigned weights that become variables in the optimization model. A norm constraint is added to sparsify the weights and determine the pruning size. The proposed mathematical model is evaluated on popular image processing datasets, CIFAR-10 and MNIST, and yields promising results.