In mineralogy and geology, identifying and classifying minerals based on their visual characteristics such as texture, color, and shape can be challenging due to the wide variety of similar looking minerals. Traditional methods often require time-consuming manual inspection by experts, which can be both subjective and prone to human error. By integrating these technologies/methods, geologists can enhance their ability to identify minerals more quickly, accurately, and consistently. In this paper, the effectiveness of computer vision for classification of visually similar minerals was evaluated. A dataset was created with special consideration to visually similar mineral groups, and was partitioned into three subsets, each increasing in number of similarity groups and number of mineral classes. A series of transfer learning models were tested. Each model was evaluated by its training and validation accuracy, and how the number of mineral classes and similarity groups impacted that accuracy. Full dataset average accuracy for tested models varied between 68.85 and 35.58%. The two models that proved to be the most effective were EfficientNetB0, which had the highest precision (68.85%) but showing overfitting, and VGG16, which had an accuracy of 43.60% with no overfitting. Lastly, across every model tested, accuracy varied significantly per class, but this variance showed no connection to visual similarity, indicating both that computer vision is not effective in the consistent classification of minerals, and that visual similarity has no negative impact on the efficacy of computer vision. This information holds great significance to the geological community, the computer science community, and industries such as mineral exploration, oil and gas, green energy solutions, and construction.