This paper develops a coupling dynamic model of a six-phase permanent magnet synchronous motor (PMSM) with a helical planetary gear system, considering the motor's dynamic behavior and the time-varying meshing stiffness of the gear system. Utilizing this model, we analyze the dynamic behaviors of the electric drive transmission system under normal operation, open-phase faults, and fault-tolerant control conditions. The research results show that the open-phase faults can increase odd-order harmonics in the phase current which introduce even-order harmonics in the torque. The torque harmonics ultimately affect the gear system dynamic behaviors with the electromechanical coupling effect, resulting in a significant increase in the amplitude of even-order harmonics in the gear system vibrations. The implementation of fault-tolerant control greatly mitigates the system's vibration response, significantly reducing the even harmonic amplitudes of torque and gear meshing force to 2%-10% of the levels observed during open-phase faults. This research provides vital theoretical insights and technical support for the reliability enhancement and optimization control of electric drive transmission systems.