<p>Existing methods for analyzing gear system dynamics fail to accurately elucidate the response mechanisms of gear surface wear—manifesting as forms such as interference friction wear and tooth surface spalling—to the dynamic characteristics of gears. Therefore, this paper proposes a novel helical gear dynamics model that incorporates pitting, wear, and friction, and has been validated through experimental testing. The Runge-Kutta method was employed to solve the system’s dynamic differential equations, yielding phase diagrams, Poincaré section diagrams, time-domain graphs, and spectral diagrams. Subsequently, the patterns of system performance variation induced by each stage of tooth surface wear and pitting evolution were investigated. The results indicate that as the cumulative number of meshes increases, multi-form damage on the tooth surfaces becomes increasingly prominent, the flank clearance of the gears widens, the decay rate of the composite stiffness amplitude accelerates, and the trend of increasing composite stiffness fluctuation amplitude becomes more pronounced. Consequently, this leads to a significant intensification of both the vibration response intensity and periodic impact effects within the time-domain diagram. In the frequency domain plot, a characteristic frequency band emerges near the meshing frequency; in the phase plot and Poincaré section diagram, the complexity of the corresponding trajectories increases, with the range of clustered discrete points progressively expanding.</p>