In 2020, an expected 476,125 individuals worldwide received a diagnosis of oral or oropharyngeal cancer. In order to find aberrant tissue changes or lesions within the oral cavity, a variety of tools and techniques are used in the detection of oral cancer. To identify cancer highly accurately, 5193 datasets consisting of 2 different classes, such as OSCC and Normal. Raw datasets might harm the model’s prediction and also it may be a cause of inappropriate feature extraction. Therefore, the dataset has been processed in many steps such as augmenting, noise reduction, smoothing, edge detection, conversion of grayscale, applying a threshold to identify high activity, applying colormap to highlight the regions, and finally combining the grayscale and red glow affected image. These procedures impact medical images and make them suitable for classification. Six models including VGG16, VGG19, ResNet50, EfficientNetB0, and InceptionV3 were employed in an automated, computer-aided approach to diagnose oral cancer by analyzing hyperspectral photographs of patients. A custom-made model was introduced and compared to the pre-executed CNN models. The best results in terms of training and testing accuracy were 99% for both Normal and OSCC detection. This paper also shows an F1 score, recall, and precision and emphasizes the use of cutting-edge technologies to fight oral cancer.