<p>Open-pit mining activities lead to heavy metal accumulation in soils, posing a serious threat to ecosystems and public health, necessitating the development of efficient, large-scale monitoring methods. In recent years, machine learning has demonstrated significant potential in uncovering complex relationships between environmental data and pollutants, offering a new paradigm for heavy metal prediction. Hyperspectral remote sensing technology, by capturing subtle spectral variations in soil properties, provides a unique data foundation for rapid, non-destructive estimation of heavy met-al content. However, extracting effective features from vast numbers of spectral bands and establishing robust predictive models remain core challenges. To address this issue, we propose a novel method that integrates fractional-order derivative (FOD) spectral preprocessing with a reindeer cyclone optimization algorithm support vector regression (RCOA-SVR) model. By leveraging exploration, exploitation, and stochastic wandering strategies, RCOA achieves a dynamic balance between global search and local refinement, thereby avoiding entrapment in local optima. Based on FOD-preprocessed hyperspectral data, the proposed model was applied to invert the concentrations of eight heavy metals (As, Cd, Cr, Cu, Hg, Ni, Pb, Zn) in 167 soil samples collected from an open-pit coal mining area in Guizhou, China, and benchmarked against partial least squares regression (PLSR) and conventional SVR models. Results demonstrated that FOD effectively enhances spectral features embedded in noisy data, with different fractional orders revealing complementary hidden information. For most heavy metals, low-order differentials (0.6–0.9) outperformed higher-order ones (&gt; 1), achieving an optimal balance between spectral information enhancement and noise suppression. When applied to heavy metal inversion, the RCOA-SVR model achieved excellent inversion accuracy, with coefficients of determination exceeding 0.94 for Cu and Hg, above 0.83 for Cr, Ni, and Pb, and consistently greater than 0.69 for the remaining elements. These findings confirm the robustness and reliability of the method for simultaneous multi-element inversion. Moreover, the RCOA-SVR model produced minimal errors at both the global distribution and individual sample levels. In summary, this study introduces an efficient and accurate framework for soil heavy metal inversion in open-pit mining areas, providing scientific support for environmental monitoring, ecological restoration, and sustainable land management.</p>