<p>In the control of horizontal vibration in rolling mills, the horizontal vibration problem restricts product quality and efficiency, and traditional models are difficult to effectively address the issue. Therefore, in order to better reflect the actual engineering situation, this study established a high-order nonlinear dynamic model that integrates the third and fifth-order nonlinear stiffnesses, in order to enhance the ability to capture the key dynamic characteristics of the rolling mill system. The specially introduced double fractional-order terms in the model are used to fine-tune the dynamic behavior, providing a more precise theoretical basis for vibration control. Firstly, in the theoretical analysis aspect, the average method is used to solve the approximate analytical solution of the system, revealing the influence of fractional differential term parameters on the amplitude-frequency characteristics; combined with the Melnikov method, bifurcation diagrams and Lyapunov exponents, the mechanism of the boundary of the attracting domain on the vibration stability is analyzed, and the chaotic critical conditions are visualized through Poincare sections, phase diagrams and time-domain graphs. Secondly, in the aspect of experimental verification, taking the cold rolling and hot rolling machines as the objects, multiple filters were used to collect and preprocess the vibration data. The time-domain responses and recurrence diagrams of the measured data and the simulation data before and after optimization were compared to verify the model's ability to accurately reproduce the vibration characteristics under different working conditions. And through optimization and error analysis, the set of solutions that best matched the experimental data was obtained.</p>