Introduction <p>The harmonic drive system (HDS) is a special type of gear transmission device, widely used in the aerospace and industrial robot industries.</p> Objective <p>This paper investigates the nonlinear dynamic characteristics and stability transition mechanism of the cup-shaped harmonic transmission system under the influence of key excitation and error-related parameters.</p> Method <p>Based on the Lagrange formula, a nonlinear torque-torsion coupling dynamic model was established, which took into account factors such as transmission errors, damping, and meshing frequency. The governing equations were solved using the Runge-Kutta method. The system dynamics were evaluated through bifurcation analysis and the maximum Lyapunov exponent. The validity of the model was verified by comparing the calculated results with the vibration spectra measured on a harmonic drive test bench.</p> Result <p>Increasing damping can enhance system stability, while increasing transmission errors will intensify instability and lead to chaotic behavior. The maximum Lyapunov exponent undergoes a sign change when the transmission error (e_m) is 0.298 millimeters, and chaotic motion occurs in the region with larger transmission errors. </p> Conclusion <p>The experimental vibration spectrum was in excellent agreement with the main frequency components predicted by the model, thereby verifying the validity of the proposed model. These research results provide valuable guidance for the design based on stability and the operational optimization of harmonic transmission.</p>