The aero-engine accessory gear transmission system is currently experiencing severe vibration, instability, and other related issues. Investigating the causes of this vibration is essential for managing system vibrations and enhancing overall stability; however, traditional research methods focusing on a single pair of gears fail to accurately characterize the dynamic response of the transmission system. In this study, we propose a comprehensive system simulation and analysis method utilizing the finite element method to examine the dynamic response of the transmission system. We also introduce a gear structure parameter optimization strategy to mitigate system vibrations, that incorporates frequency modulation and vibration avoidance techniques. A vibration control study of the gear transmission system revealed that during operation, the vibrations of the Z2 and Z3 gears are the most pronounced. By implementing frequency modulation and vibration avoidance technologies, the web thickness of the Z2 and Z3 gears was increased to 9 mm and 6 mm, respectively, resulting in a resonance margin exceeding 10%. After optimizing gear structure parameters, the modal frequencies of Z2 and Z3 gears are 16706.10 Hz and 16019.20 Hz respectively. Notably, the total mass of the system increased by only 0.07 kg, effectively controlling both the mass and vibration of the transmission system.