Fruit maturity classification is crucial to meet the high-quality food production demand on a large scale. Manual fruit categorization based on their maturity level is a labor-intensive, time-consuming, and tedious process. This study introduces a novel, automated computer vision-based approach for maturity classification, ensuring consistent quality and streamlined production. The proposed FRMC-Net (FRuit Maturity Classification Network) leverages the pre-trained MobileNetV3 model, which is fine-tuned using transfer learning for feature extraction. The extracted features are further refined by incorporating the shuffle attention module in the FRMC-Net. The shuffle attention module integrates the channel and spatial attention maps, shuffling the channel groups to emphasize the most important feature maps. A publicly available guava dataset containing digital and thermal images is used for experimental results, which contain three different categories, namely half mature, immature, and mature. The classification performance of the FRMC-Net has been compared with existing state-of-the-art CNN models. The FRMC-Net surpasses the existing models by obtaining the highest classification accuracy (96.90% & 95.89%) on thermal and digital images.