Deep neural network training has demonstrated its extraordinary capabilities in recent years, outperforming various machine learning methods. Optimizing the weight parameters of deep neural network training is a time-consuming process due to its computational complexity. Gradient descent (GD) is currently a widely used technique for optimizing weight parameters in conjunction with backpropagation (BP) techniques, but limitations such as slow convergence and tendency to local minima have been widely pointed out. In differential-based techniques, many problems are characterized as non-convex, discontinuous, and non-differentiable. This leads to the gradient vanishing problem. To address these challenges, an alternative optimization approach is explored by utilizing meta-heuristic algorithms (MH). The general form of the MH algorithm struggles to maintain the variation between descendants, which can lead to premature convergence of the solution. Furthermore, the computational overhead of the MH algorithm can make the optimization of high-dimensional problems inefficient. For this reason, the application of a new MH-based method is explored to optimize the weight parameters of deep neural network training. Furthermore, this paper aims to highlight the inherent shortcomings of BP and GD techniques and the general MH algorithm.