Dental microwear texture (DMT) analysis is a critical proxy for reconstructing the diets of extant and extinct mammals. While craniodental morphology reflects selective pressures across evolutionary timescales, microwear captures localized, short-term dietary signals over weeks to months. However, the high dimensionality of modern 3D surface microtexture data, often spanning disparate parameter sets (such as ISO standards and scale-sensitive fractal analysis, SSFA), complicates classification, particularly when working with limited paleontological datasets. To address this, we present a robust machine learning pipeline designed to automatically classify primate samples (\(N = 99\)) across 6 dietary groups and 7 species. Our methodology leverages a nested leave-one-out cross-validation framework to evaluate multiple classifiers, including multinomial logistic regression (MLR) with various regularization approaches, Naive Bayes, and tree-based ensemble algorithms (e.g., Random forests, XGBoost). Our results demonstrate that Lasso-regularized MLR and Naive Bayes yield the highest predictive performance while enforcing strict feature selection to maintain interpretability. Crucially, models relying exclusively on ISO parameters consistently outperformed those using SSFA, as ISO variables better capture the microroughness localized mechanical abrasions generated by specific diets in our dataset. Furthermore, the integration of novel Fourier-based descriptors and isotropy variables significantly enhanced models’ discriminating power. By providing a mathematically rigorous framework to isolate precise ecological signals from noisy, high-dimensional data, this approach enables more accurate and reproducible dietary classifications. Ultimately, refining these dietary reconstructions is essential for resolving broader questions regarding niche partitioning, species evolution, and paleoecological dynamics.
