<p>Chatter is an undesirable vibration that negatively affects surface quality and machining efficiency. Detecting chatter at an early stage can improve the performance of high-speed milling, but difficult to achieve. Therefore, efficient techniques for online monitoring are needed to address the issue of chatter monitoring. To this end, this paper proposed an improved variational mode decomposition (VMD) online chatter detection model based on the Kepler optimization algorithm and composite fitness function, which is used to extract chatter-sensitive features from complex signals and achieve automatic detection of chatter based on energy entropy. Used a simulated signal with chatter frequency and established three groups of high-speed milling experiments to evaluate the effectiveness of the proposed method. The results indicate that the online monitoring method based on improved VMD has significant advantages in extracting chatter-sensitive features and recognizing chatter state online.</p>