<p>Subspace training (ST) recently emerged as a powerful technique to theoretically analyze and practically improve the training of deep neural networks (DNNs). First proposed as a tool for analyzing objective landscapes, it has now been found valuable for practical tasks such as model compression and defense against overfitting and adversarial attacks. The key mechanism behind ST is to train a DNN in a <i>low-dimensional subspace</i> by projecting back-propagated gradients onto the subspace. This work sheds light on ST from a new perspective; we theoretically show that ST can mitigate an optimization issue regarding gradient noise. To showcase this novel aspect of the ST, we incorporate ST into the normalized stochastic gradient descent (NSGD), an optimization technique that, although widely used, suffers from gradient noise. The experimental results on visual classification tasks using CNNs trained by our ST+NSGD demonstrate empirical efficacy of the approach, supplementing our primary theoretical analysis.</p>