Activation functions play a pivotal role in the performance and convergence of deep neural networks (DNNs) as well as convolutional neural networks (CNNs). In this study, a comprehensive comparative analysis is conducted on fourteen activation functions, including both classical functions, for example, ReLU, Sigmoid, Tanh and recent proposals such as SiLU, Mish, Serf, Nipuna. We apply these functions to a DNN architecture trained on the MNIST dataset, and then on a CNN trained on both MNIST and CIFAR-10. Results include a-ccuracy, training time, inference speed, and the cumulative loss area (computed via numerical integration). The results indicate that while some functions deliver high accuracy and an efficient convergence, others result in statistically significant decreases in accuracy and prolonged convergence times. This paper outlines the methodology, provides the mathematical framework, and presents and discusses the findings.