The microscopic parameters of the helical gear tooth surface directly affect the meshing performance and dynamic transmission characteristics of helical gear pairs. Therefore, this paper takes a modified turbine engine dry box as the research object and establishes an analytical model for nonlinear coupling excitation of mesh stiffness and error and a hybrid dynamic model of a helical gear transmission system considering the modification. Firstly, by combining the helical meshing principle and the meshing property of the helical gear face, the calculation method of helical gear meshing line position and length in three-dimensional space is improved. According to the principle of force and deformation decomposition and the coupling relation of stiffness and error, an analytical calculation model of modified helical gear time-varying meshing stiffness considering axial deformation was proposed. Secondly, a hybrid dynamic model including shaft segment elements, meshing elements, bearing elements, and box elements is established according to the node finite element method and substructure theory. Finally, the correctness of the excitation model and the dynamic model was verified by the finite element method and experiments, and the influence of different modification methods and modification amounts on mesh stiffness, transfer error, and vibration response characteristics of the system were analyzed. The optimal modification parameters were obtained with the minimum vibration acceleration as the optimization objective. The results show that the average meshing stiffness error calculated by the analytical method is only 0.278%, and the maximum error between the comprehensive meshing stiffness and the finite element method is only 3.554%. Under the optimum modification parameters, the experimental results show that the vibration intensity meets the requirements of the gearbox.