<p>Atom Search Optimization (ASO) is a meta heuristic algorithm developed based on the principles of molecular dynamics. Although it has successfully solved many engineering problems, it still suffers form premature convergence and poor precision in some cases. Therefore, this paper is devoted to proposing a hybrid atom search optimization algorithm named HMASO to improve convergence accuracy and speed. In HMASO, a new acceleration update strategy is employed to better balance the exploration and exploitation. An enhanced local search strategy with Lévy flight and spiral search mechanism is employed to boost the convergence accuracy, and a perturbation strategy is adopted to improve the potential of the algorithm to jump out of local optima. Meanwhile, a nonlinear convergence factor is designed to control the convergence speed. The results of the qualitative analysis of the population diversity and search processes of ASO and HMASO suggest that HMASO achieves a better balance between exploration and exploitation, and providing more opportunities to escape from local optima. To verify the performance of HMASO, it was compared with ASO, two variants of ASO and six newly developed algorithms on the CEC2017 benchmark functions. Experimental results in 50 and 100 dimensions show that HMASO significantly outperforms the other compared algorithms in terms of of mean, best value, and speed of convergence. Boxplots confirm that the HMASO algorithm has better stability. The overall performance of HMASO is significantly better than comparison algorithms. The results of the Wilcoxon signed rank test and the Friedman test further statistically confirm the superiority of HMASO. Finally, we apply the HMASO algorithm to the complex optimization problem of feature selection for quantitative quantitative structure–activity relationship (QSAR) models in chemoinformatics. We used HMASO to optimize the input features of a multiple linear regression model to build a predictive model for the permeability of Caco-2 cells. The developed model containing 124 features achieved the best prediction accuracy of <i>R</i>²_CV5 = 0.7664. The results show that HMASO can select the most effective features compared to comparison algorithms.</p>