Scene Text Image Super-Resolution with CLIP Prior Guidance
摘要
Scene Text Image Super-Resolution (STISR) plays a crucial role in enhancing text readability within natural scenes, impacting OCR systems, visual question answering, and image retrieval. Existing STISR methods often fall short, either neglecting textual information entirely or utilizing it ineffectively. We attempt to bridge this gap with a novel two-fold approach. Firstly, we use CLIP (Contrastive Language-Image Pre-Training), a powerful model that can map both images and text into a shared latent space, enabling it to assess image-text alignment. We harness CLIP’s ability to understand the semantic relationship between images and text. By incorporating CLIP-generated image representations that capture these inherent textual features, we effectively guide the super-resolution process, leading to more accurate reconstructions. Secondly, we propose a novel TrOCR (Transformer-based OCR) loss function to supervise the super-resolution process from a text-centric perspective. Our loss function enforces consistency between the super-resolved output and the high-resolution ground truth image in terms of their text content. Experiments conducted on the benchmark TextZoom dataset demonstrate that our approach not only improves visual quality but also boosts text recognition accuracy.