Tahitian pearls are globally known for their luster, rainbow-like color palette, and various shapes. As pearl trading is French Polynesia’s second-largest revenue source after tourism, the manual classification remains labor-intensive, with shape evaluation consuming 60–70% of the expert workload. Furthermore, subtle shape variations make evaluation subjective, often requiring multiple experts to ensure consistency, increasing costs and time. This study introduces a scalable, automated pipeline for pearl shape classification. The system integrates a mirror-based multi-view imaging setup to capture comprehensive views, YOLOv8n for precise object detection, and SAM for segmentation. Discriminative handcrafted features are selected through NSGA-II and combined to enhance classification performance. XGBoost performs a two-step hierarchical classification using selected discriminative handcrafted features. First, pearls are categorized into broad classes (asymmetric, circled, symmetric), then the classification of symmetric pearls is refined into sub-classes like button, oval, etc. Trained on a dataset of 3,929 pearls labeled by five experts, the system achieves weighted F1 scores of 95% and 75% for steps one and two, respectively. Multi-view data was proven crucial for capturing subtle shape variations, particularly for the fine-grained step two classification. Compared to deep-learning-based models, the pipeline demonstrates superior cross-validation stability and accuracy. This framework provides a scalable and objective solution for pearl classification, significantly reducing expert workload while improving consistency. Beyond pearls, the methodology could be used for other applications, such as jewelry or gemstone assessment.