A crystal is then cut down into pieces and, when polished, becomes a gemstone. Classification of gemstones helps in accurate identification and differentiation of various kinds of gemstones. The physical and optical traits of any gemstone help in their correct identification. Nowadays, the worth of any product is measured on the basis of their market value and gemstone’s classification plays a vital role in deciding the market value of gemstones. Proper gemstone’s classification contributes in establishment of a market price that is fairer. It is a difficult task to identify and classify gemstones due to the same find of nature, variations and colours. This paper consists of a deep learning approach that helps to classify gemstones using Transfer Learning with MobileNetV2 along with data augmentation and fine-tuning. The dataset consists of separate training and testing data, with more than 3200 images categorised into 87 classes. There are several images in variable size in.jpeg format. The model takes an image as an input and predicts its class, the model proposed archived an accuracy of 74% positively.