As convolutional neural networks (CNNs) are increasingly used for computer vision tasks, the environmental consequences of their high energy consumption cannot be denied. CNNs especially require a lot of computational power due to their high dimensional inputs, their high number of parameters, and their large size. Existing literature introduces techniques that do not specifically focus on energy consumption as such, but rather on improving the performance, portability, or speed of CNNs. These approaches mostly focus on the testing phase. Furthermore, the inconsistency of metrics that are used for estimating or measuring energy consumption reduces the comparability of these techniques. To address the gap in existing techniques that largely overlook energy consumption, this work proposes a novel approach, T-SE, aimed at reducing the energy usage of CNNs during both the training and inference phases. This new method leverages Squeeze-and-Excitation networks [7], introducing a threshold in this architecture to reduce the number of computations performed by the CNN, and in turn the energy consumed by it. Results show that more than 500,000 convolutions per input (equivalent to around 13% of the total convolutions in the CNN) can be avoided without decreasing significantly the performance of the CNN.