<p>This study investigates the nonlinear dynamics of a two-degree-of-freedom (2-DOF) rotor active magnetic bearing (AMB) system, incorporating geometric coupling and nonlinear effects. Specifically, the complex dynamic behaviors arising from the combined action of 1:2 internal resonance and 1:2 superharmonic resonance are elucidated. Approximate analytical solutions are first derived via the method of multiple time scales. Subsequently, the local response, global stability, and chaotic evolution paths are systematically explored utilizing the cell mapping method, 3D largest Lyapunov exponent spectra, and bifurcation diagrams. Results indicate that the amplitude-frequency response exhibits significant anisotropy: the horizontal and vertical degrees of freedom manifest hard-spring and soft-spring characteristics, respectively. Furthermore, external excitation and damping parameters are found to primarily regulate the generation and topological merging of Isolated Response Curves. Global stability analysis reveals that as the external excitation amplitude increases from 0.18 to 0.48, the boundaries of the basins of attraction undergo fractal erosion, resulting in a sharp decline in the system’s robustness against perturbations. The negative damping region is identified as a high-probability zone for chaotic motions, with period-doubling bifurcation serving as the primary instability mechanism leading to chaos. This study clarifies the evolution mechanisms of nonlinear vibrations under multi-parameter coupling, providing theoretical guidance for the optimal design and stable control of rotor-AMBs system.</p>