<p>In the actual measurement process of the new gear grinding machine measuring mechanism, due to the existence of the trigger probe measuring force error, the on-line measuring accuracy of the gear grinding machine is extremely affected. The existing research predominantly concentrates on establishing a bending deformation model for the measuring rod or compensating for errors using the relationship between measurement error and measuring force. However, few studies have taken into account the equivalence between measuring force error and the center position error of the probe. This paper proposes a novel and simplified method for compensating the measuring force error of the probe in accordance with innovative measuring mechanism. Based on the tooth profile measurement principle, the measuring force error is transformed into the center position error of the probe. The Non-dominated Sorting Genetic Algorithms-II (NSGA-II) is utilized to calculate the optimal probe center position error for left and right tooth flanks. Measuring the tooth profile of the left and right tooth flanks, the center position error of the probe on the left and right tooth flanks is compensated respectively. An experiment involving the on-line measurement of three sections of tooth profile is conducted by using a self-developed Computer Numerical Control (CNC) gear grinding machine. Results indicate that prior to compensation, the tooth profile accuracy of the left flank is level 5, while that of the right flank is level 6. Post-compensation, the tooth profile accuracy improved by one to two levels, stabilizing at approximately level 4, which aligns with Gleason’s tooth profile measurement result. These findings validate the accuracy of the proposed error compensation method and provide valuable insights for enhancing the on-line measurement accuracy of gear grinding machine.</p>