Aiming at the problem of dynamic characteristics change caused by the change of micro-tooth surface morphology in the operation of gear system, a nonlinear dynamic modeling method considering micro-tooth surface morphology is proposed. The surface topography of the gear after three different working hours was measured, and the data of the arithmetic average deviation, fractal dimension and fractal roughness contour are obtained. Based on the fractal theory, the tooth clearance、the contact stiffness and the time-varying meshing stiffness of the gear is corrected. The comprehensive transmission error excitation is mathematically described, and the calculation formulas of the friction torque direction coefficient and the force arm are derived. Combined with the mixed elastohydrodynamic lubrication model, the effect of micro-tooth surface morphology on oil film characteristics is revealed, and a nonlinear dynamic model with time-varying friction coefficient and oil film damping is established.