<p>The rotor-bearing-support system is widely used in engineering applications for energy conversion. Due to factors such as bolted joints and structural complexity, the coupling vibration characteristics of the system are usually complex, leading to serious stability issues. Under working conditions, bending and torsional vibrations often occur concurrently with coupling effects, especially in large rotor systems. This study aims to analyze the stability of the bending-torsional coupled vibration of a bolted joint rotor-bearing-support system. Taking into account the lateral, torsional, and bending vibrations of the rotor, along with the relative motion of the bolted joints, the governing equations for the rotor system are derived using the energy theorem and Lagrange’s principle to investigate the stability of the system. Through the numerical integration method, the vibration characteristics of the system are analyzed, and the effects of nonlinear bending stiffness and disk eccentricity on the stability of bending and torsional motions are studied. The research findings indicate that the bending stiffness of the bolted joint and disk eccentricity have a significant effect on the bending and torsional vibration stability. The present work has implications for the design of rotor-bearing-support systems in engineering practice, providing guidance for improving the stability of such rotor systems.</p>