<p>The spline connection is a prevalent mechanical transmission method widely used in the automotive, aviation, and heavy machinery industries. It is characterized by its simplicity and reliability, playing a crucial role in the overall performance of transmission systems. However, poor lubrication conditions can lead to elevated internal friction torque within the spline, which may induce self-excited vibrations in the rotor system. For rotor systems connected by splines, determining the global boundary of self-excited vibration response is essential for preventing system failure and optimizing design. Current research lacks a comprehensive analytical method to delineate this boundary. This study aims to derive the self-excited vibration boundary for a spline-connected rotor system using an analytical approach. First, a four-degree-of-freedom extended Jeffcott rotor model is constructed. The validity of this model is verified by comparing its accelerated response process to that of a finite element model. Finally, utilizing Lyapunov’s first method, this paper derives a detailed analytical expression for the boundary of the system’s self-excited vibration response based on the extended Jeffcott rotor model. This research provides valuable theoretical support for the reliability and stability design of spline-connected rotor systems.</p>