<p>Effect of the preload of the bolt and elastic deformation of the spigot on the dynamic behavior of the bolted flange is usually neglected in the traditional models. Furthermore, there are limited studies on the vibration characteristics of the rolling bearing and bolted rotor system with spigot. In this paper, an analytical model of the bolted flange with spigot is proposed. The spigot is simulated by a rotational spring and the dynamic behaviors at the spigot interface are simulated by the Jenkins element. Moreover, the preload and deformation of the bolt are considered. The results under axial load and harmonic load of the proposed model show good agreement with the results of the three-dimensional (3D) finite element (FE) simulation. To improve the calculation efficiency of calculating the dynamic behavior at the interface, a Gaussian process (GP) model is constructed to predict the tensile load under various displacements and preloads. Then, a nonlinear vibration analysis of a rolling bearing-bolted rotor system with a spigot is conducted. Detailed parametric analyses under small and large unbalance loads, including interference fit, preload of the bolt and number of the bolt, are conducted to investigate the vibration characteristics of the rotor system. Results show that the bolted joint with spigot mainly produces stiffness loss and damping at the interface. Under large unbalance load, the dynamic characteristics are affected significantly. Moreover, the first two critical speeds can be increased by increasing the interference fit, preload of the bolt and number of the bolt.</p>