Dental caries is one of the most prevalent and painful infectious diseases affecting the oral cavity. Early detection of carious lesions is crucial in preventing the infection from spreading. While dentists commonly rely on X-ray images to identify these lesions, the low intensity of dental X-rays often makes it challenging to pinpoint the exact location of the affected areas. Additionally, the shortage of dentists in government hospitals further complicates the timely treatment of large patient volumes. This study proposes a system designed to assist dentists in quickly and accurately detecting carious lesions. Deep learning methods are not ideal for this application due to the absence of a sufficiently large training dataset to create a robust pre-trained model. As a result, traditional handcrafted methods are preferred in this case. Dental X-rays capture not only the teeth and bone structures of the jaws but also the soft tissues within the gum regions. Consequently, standard texture-based segmentation techniques are insufficient for detecting caries lesions. These lesions typically manifest as catchment basins, with the greatest depth at the center. To model this characteristic, the isophote and geodesic active contour methods are particularly effective. However, accurately locating the suspected caries region requires a multistage background elimination process. The first stage of this process involves calculating randomness and rescaling that value based on a small training dataset. Background elimination is then performed using a modified k-means clustering approach on the entropy and grayscale values of the X-ray image. The number of clusters is determined automatically by analyzing the distribution of data points, ensuring that the technique avoids over-clustering. Since most carious lesions are found within the teeth region and are surrounded by other teeth, this spatial relationship is also considered to further eliminate the background and detect suspected lesions. Due to the limited availability of online dental X-ray databases containing caries lesions, the “Digital Dental Periapical X-ray Database for Caries Screening” was used to test the proposed method. The system achieved an accuracy of 94% with an average computational time of under 4.5 s. This method offers a viable alternative for detecting regions of interest (ROI) in situations where deep learning approaches are hindered by a lack of comprehensive training data. However, the method’s effectiveness diminishes when working with low-resolution X-ray images. Low resolution can lead to confusion between randomness and noise, making it difficult to accurately identify the characteristic properties of the caries lesions, thus impairing detectionRescaling,Isophote accuracy.