The early detection of Oral Potentially Malignant Disorders (OPMDs) is critical for successful intervention and improved patient outcomes. Screening for potentially malignant oral lesions is crucial before classifying them into different classes and subclasses of OPMDs. This study compares two transformer-based segmentation models, Med-SAM and SegFormer-B5, with CNN models to detect OPMDs in the oral cavity from photographic images. Using a dataset of 1,435 images from 486 patients, the models were evaluated with four scenarios by incorporating different image types for training and testing data, viz. full images and cropped images. Three metrics were used to assess the model: Percentage overlap with the actual lesions, F1 score, and IoU. The SegFormer-B5 model varied in performance, achieving its best results with an F1 score, IoU and percentage overlap of 0.83, 0.72 and 0.83, respectively, when training and testing on cropped images. When trained and tested on full images, the SegFormer-B5 model achieved an F1 score of 0.59, IoU-0.45, and percentage overlap of 0.67. On the other hand, the Med-SAM model demonstrated moderate performance on full images with an F1 score of 0.38, a percent overlap of 0.73, and IoU-0.26 but significantly excelled with cropped RoIs (Region of Interest), reaching an F1 score of 0.73 and percentage overlap of 0.82 and IoU-0.59. The high F1 scores, IoU, and percentage overlap achieved by these models underscore their capability to screen OPMDs effectively and support the classification models by enhancing the precision and reliability of OPMD diagnostics.