Quantization is an effective strategy for model compression, facilitating the efficient deployment of deep neural networks on resource-constrained edge devices. However, existing post-training quantization methods often fail to differentiate the significance of weights in the quantization process, which can lead to considerable information loss and adversely affect the model’s inference capability. To address this, We propose a Gradient-aware Incremental Network Quantization method (GaINQ) that utilizes weight gradients to evaluate parameter significance, and accordingly guides the quantization process to achieve dynamic quantization of model parameters. Additionally, we incrementally apply quantization and integrate a nearest power-of-two approach to further compress the bit width required for weight representation. This strategy enhances storage efficiency and replaces complex arithmetic operations with shift operations during inference, thereby improving computational efficiency. When applied to the GoogLeNet model, GaINQ demonstrates a notable improvement over state-of-the-art quantization methods, with accuracy gains ranging from 0.6% to 4.39%, achieving an accuracy of 96.25%.