<p>Slope instability is a prevalent dynamic disaster encountered in the construction of geotechnical engineering projects. Intelligent detection and early warning systems serve as crucial measures for preventing and controlling slope instability. To accurately and efficiently predict the stability state of slopes, we propose a combined model that integrates the Newton–Raphson optimization algorithm (NRBO) with an optimized extreme gradient boosting tree (XGBoost). Firstly, the primary factors influencing slope instability are thoroughly analyzed. The sample outline is standardized utilizing polar deviation, and the distribution of sample classes is balanced through the application of the Synthetic Minority Over-sampling Technique (SMOTE). Secondly, the XGBoost model is optimized by fine-tuning parameters such as maximum depth (max_depth), learning rate (learning_rate), subsample rate, column sampling rate (colsample-bytree), and minimum loss (gamma) through NRBO. The stability of the model was assessed using a ten fold cross validation method, while the prediction results were comprehensively evaluated utilizing metrics including accuracy (<i>Acc</i>), precision (<i>Pre</i>), recall (<i>Rec</i>), F1 score (<i>Fs</i>), and Cohen’s Kappa coefficient (<i>Ka</i>). Finally, the SHAP additive interpretation method is employed to elucidate the significance and contributions of features influencing the XGBoost model. This model is subsequently applied to ten specific engineering case studies. The results show that after NRBO optimization, the optimal values for the maximum depth, learning rate, subsample proportion, column sample proportion, and minimum loss of the XGBoost model are 7, 0.8247, 0.6326, 0.6263, and 0.0758, respectively. Based on the SHAP model analysis, the main factors influencing the stability of the slopes are <i>g</i>, <i>c</i>, <i>φ</i>, <i>H</i>, and <i>j</i>.</p>