Considering the contact condition of the modified gear pair, a method for calculating the time-varying meshing stiffness (TVMS) and friction excitation of modified gear pair is proposed, and then the TVMS and friction excitation of the gear pair before and after modification are analyzed. Taking into account of the TVMS, friction excitation, mesh impact, mesh phase relationship and manufacturing errors, a dynamic model of the planetary gear transmission system is established, and the dynamic performance of gear system before and after modification is compared. In order to minimize the sum of root mean square of torsional vibration velocity of the gear system, a genetic algorithm is used to optimize the modification parameters. After optimization, the vibration condition of the gear system has been significantly improved. Reasonable 3-D tooth modification can enhance the vibration and noise of gear systems.