In this paper, the nonlinear dynamic meshing model of the planetary gear with the flexible shaft deformation was derived in detail, and nonlinear dnamic coupling forces between the rigid gear pairs and the flexible shaft were analyzed. Then, the nonlinear hybrid dynamic model of a single-stage planetary gear transmission system was established, and nonlinear lumped-parameter vaiables of the rigid gear and nonlinear distributed-parameter variables of the flexible shaft were taken into account in this model. Nonlinear lumped paramter variables of the rigid gear include the time-varying center distance, the time varying backlash, the dynamic meshing stiffness and the dynamic meshing force, and nonlinear distributed-parameter variables of the flexible shaft include the mass, the stiffness, the elastic force and the gravity. In addition, the distributed-parameter model of the flexible shaft was obtained by using the three-dimensional two-node nonlinear beam element described by the Absolute Node Coordinate Formulation (ANCF), and the lumped-parameter model of the rigid planetary gear was obtained by using the lumped-parameter method of the gear dynamics. Furthermore, the nonlinear hybrid dynamic equation of a single-stage planetary gear transmission system was derived by combining the lumped-parameter model of the rigid planetary gear and the distributed-parameter model of the flexible shaft, and the equation was solved by the generalized - α numerical method. Finally, the complex nonlinear dynamical behaviors and the vibration characteristics of the system were analyzed, and the rigid-flexible coupling dynamic mechanism between the meshing of the rigid gear and the deformation of the flexible shaft was revealed.