<p>Rock layer subsidence in subsea gold mines is critical for the safe mining of subsea areas. To accurately predict the rock layer subsidence of the subsea gold mine, 152 sets of rock layer subsidence data were collected on-site by monitoring at the Sanshandao gold mine. Twelve influencing factors were selected as input variables, including days, drilling depth, tunnel depth, tunnel angle, density, Poisson's ratio, compressive strength, elastic modulus, tensile strength, softening coefficient, cohesion force, and internal friction angle, and the rock layer subsidence displacement was chosen as the output variable. Furthermore, an independent Extreme Gradient Boosting (XGBoost) model was considered, and four hybrid models (WOA-XGBoost, SOA-XGBoost, GWO-XGBoost, and XGBoost) were development. The coefficient of determination (<i>R</i>²), root mean square error (RMSE), and mean absolute error (MAE) were obtained for the four models. The results showed that the WOA-XGBoost model was the most reliable model (<i>R</i>² = 0.983). Additionally, according to the feature importance analysis, the softening coefficient, Poisson's ratio, and tunnel depth were identified as key factors for the rock layer movement subsidence. Finally, the WOA-XGBoost model was applied to predict the rock layer movement subsidence in the Sanshandao gold mine, and engineering verification was conducted. Meanwhile, it can also provide a new approach for the prediction of the rock layer movement displacement subsidence in subsea gold mines.</p>