Text spotting has received attention due to several real-world applications. Most existing spotting methods use textual information to solve text-spotting challenges and ignore non-text information (background part). Therefore, the current text spotting methods lack robustness and generalization. Here, we propose an approach that introduces a multimodal deep learning framework for classifying similar (interrelationship between scene text and non-text) and non-similar (no relationship between scene text and non-text) images to improve the text spotting performance. For an input scene text image, DeepSolo, which is a state-of-the-art text spotting method, has been used to segment text from non-text regions. The text regions are fed to the Google OCR for recognition results. Consequently, the recognition results are fed to the generative model (Fooocus) as a prompt for generating an image. It is expected that the generated image should be similar to the non-text region of the input image. This is because we believe that text and background have a strong correlation. The visual features from the generated image and input images are extracted using the combination of ViT and EfficientNet. The labels are generated for the input image and the generated images using the Google Vision API. The BERT is explored to extract textual features from the labels. The visual and the textual (semantic embedding) are combined using a bilinear fusion to classify similar and non-similar images. The proposed classification is validated and compared with the existing methods to demonstrate its promising performance. The state-of-the-art text spotting methods are tested before and after classification to show the effectiveness of the classification. This is the first work to classify similar and non-similar images using a diffusion model to improve text spotting performance.