One of the main challenges of zero-trust systems is the effectiveness of authentication methods. To enhance the accuracy of authentication, this paper proposes an authentication method based on an improved butterfly optimization algorithm and XGBoost. This method first uses multiple strategies to improve the butterfly optimization algorithm to obtain MSIBOA, then applies MSIBOA to XGBoost for hyper-parameter tuning to obtain MSIBOA-XGBoost, and finally applies MSIBOA-XGBoost to user authentication. Therein, MSIBOA introduces dynamic switching probability to balance local and global search, Gaussian perturbation to improve global search capability and adaptive weight to enhance local search capability. The results show that MSIBOA outperforms the compared heuristic optimization algorithms in terms of search accuracy and convergence speed of parameter optimization. In addition, in terms of verification effectiveness, compared with each classification method, the proposed MSIBOA-XGBoost improves the Accuracy, Recall, Precision and AUC metrics by an average of 1.74%, 2.39%, 2.22% and 3.01%, which shows that the proposed method enhances the accuracy of authentication and can be better adopted to the zero-trust system.