<p>American ginseng is often substituted with visually similar herbs by unscrupulous merchants due to its high value. This study introduces an RGB image-based computer vision model, which combines self-supervised pre-training and supervised fine-tuning, enabling the rapid, non-destructive identification of American ginseng. To overcome the limitations of labeled data, a feature extraction model was pre-trained using 5000 unlabeled herb samples with the Momentum Contrast (MoCo) framework. Subsequently, six supervised learning methods for fine-tuning were evaluated and fully connected (FC) with full fine-tuning was determined as the optimal classifier. The proposed model, HerbMoCo, outperformed traditional models trained on limited labeled data. t-SNE and Gradient-weighted Class Activation Map (Grad-CAM) highlighted its robust feature extraction and interpretability. Additionally, a user-friendly Graphical User Interface (GUI) was developed to assist non-experts in identifying American ginseng and similar herbs. This method has been validated for environmental sustainability through evaluations of greenness and whiteness. It provides an efficient and robust solution for herb identification, especially in scenarios where there is limited labeled data and an abundance of unlabeled samples.</p>