Because DNNs are complex nonlinear models, solving for the optimal model parameters analytically (i.e., take gradient and find minimum) is not possible. Instead, we must rely on iterative numerical optimization methods to get an approximate of the optimal solution. In this section, we cover loss functions and gradient descent optimization, which are the fundamental concepts that allow a network to learn. We also cover a variety of tweaks that we can make to the gradient descent algorithm that enable better performance.