Purpose <p>The planetary gear transmission system is a key propulsion component in Tunnel Boring Machines (TBMs). Traditional three-stage 2&#xa0;K-H planetary gears suffer from severe load deviation, causing vibration and noise. This study proposes a double compound planetary gear, focusing on its load-deviation suppression performance and nonlinear dynamic characteristics to reduce system vibration and noise.</p> Method <p>First, force balance models of the traditional 2&#xa0;K-H planetary gear and the compound planetary gear were established, and the load-deviation suppression mechanism of the compound planetary gear was examined through comparative analysis. Then, a nonlinear dynamic meshing model was then developed and solved using the variable-step fourth-order Runge–Kutta method. Bifurcation and maximum Lyapunov exponent diagrams were used to examine system dynamics under varying conditions.</p> Results <p>Comparative analysis based on the force balance model shows that the proposed double compound planetary gear structure only bears unidirectional torque and demonstrates good load-sharing performance. In addition, the results indicate that increasing temperature and damping ratio facilitates the transition of the system from chaotic motion to periodic motion. When the fluctuation of the error amplitude is sufficiently small, the system can also maintain stable periodic operation.</p> Conclusion <p>The proposed double compound planetary gear structure can effectively alleviate the load-deviation problem in TBM power transmissions. Increasing temperature and meshing damping ratio can suppress system vibration. These findings are valuable for enhancing TBM operational reliability and guiding future planetary gear design and applications.</p>