Millimeter-wave (mmWave) communication technology with high throughput and high reliability attracts much attention in both academic and industrial fields. Localization of mobile mmWave communication devices can direct mmWave beam steering and avoid the beam alignment overhead. However, existing approaches for commercial mmWave communication devices suffer the channel state fluctuation and can only work on static devices. To provide accurate localization for mobile mmWave communication devices, we propose NNLoc, a neural network-based approach for enhancing mmWave localization accuracy. We first analyze the feasibility of classifying sensing data into different qualities. Then we conduct a CNN architecture for filtering out the sensing data with high qualities. The experiments on COTS mmWave communication devices show that NNLoc can increase the localization accuracy significantly and reduce the median localization error of mobile devices from 39.23 cm to 6.39 cm with only single items of CSI measurements.