The current era is characterized by an unprecedented explosion of data, alongside significant advancements in hardware technologies. The development of diverse Graphics Processing Units (GPUs) and the parallelization of computing tasks have catalyzed notable breakthroughs in fields such as Image Processing and Computer Vision. These technological innovations have fostered the widespread adoption of Deep Learning globally. Central to these advancements is the Backpropagation algorithm, which leverages the Chain Rule to enhance the learning processes in neural networks with the aim of minimizing learning loss. However, the efficacy of Backpropagation can be substantially improved through the use of various optimization algorithms. The selection of an appropriate optimization technique remains a critical and complex decision for Deep Learning developers, as the optimal choice varies based on specific application requirements and system constraints. This variability highlights the necessity for customized optimization to develop robust and efficient Deep Learning systems. This study provides a metrics-based, comprehensive experimental comparative analysis of three widely-used optimizers-Stochastic Gradient Descent (SGD), Adaptive Moment Estimation (Adam), and Root Mean Squared Propagation (RMSprop), utilizing the PyTorch library. Furthermore, we propose a 7Es’ model to offer Deep Learning users a simplified method for a holistic evaluation of models’ metrics during training. The aim of this work is to equip Deep Learning practitioners with the insights necessary to make informed decisions regarding optimizer selection, by elaborating on the theoretical foundations, practical implementations, and comparative performance of these optimizers.