In order to improve the working performance of electric vehicle gearboxes, taking the electric vehicle reducer as the research object, combined with the second-generation genetic algorithm, and taking the reduction of vibration acceleration of the gear transmission system as the optimization objective, a gear modification method considering the nonlinear factors is proposed. To perform modal analysis and order analysis, a rigid-flexible coupling model considering time-varying meshing stiffness, meshing impact, and tooth-face friction excitation is established, and the microscopic trimming parameters of the original design are imported into the model. Then, with the peak-to-peak gear transmission error and the linear loading as the optimization objectives, the micro-shaping parameters of the gears are optimized through genetic algorithms. The simulation results show that the optimized gear transmission model reduces both the peak-to-peak transmission error and the linear loading. Specifically, gearbox vibration has reduced the vibration acceleration by 18.74% under the nominal working condition, and the meshing force distribution has become more uniform, which improves the meshing performance of the gears and ensures the stability of the reducer operation. The gear modification method proposed in this paper has practical engineering significance in reducing the vibration for the gear transmission system housing.