Image super-resolution is a technique aimed at enhancing the resolution of images and is widely utilized in fields such as remote sensing and medical imaging. We view image super-resolution as a reconstruction problem for inferring missing structural information from incomplete observations. This perspective highlights the need for methods that do not merely generate pixel-level corrections, but instead analyze and infer latent geometric and semantic patterns embedded in multimodal data. In this work, we present PromptFusionSR (PF-SR), a novel framework for text-assisted image super-resolution. In contrast to conventional approaches that rely solely on visual features, our method leverages the BLIP3 vision language model to extract contextual prompts-short textual descriptions-directly from low-resolution images. A periodic semantic–structural fusion mechanism enables the model to integrate complementary information channels, enhancing its ability to interpret high-frequency patterns under severe degradation. These textual and information cues guide a pretrained stable diffusion model for 4 \(\times \) upscaling, yielding high-resolution outputs with improved fidelity and richer visual textures. This work further explores how complementary information channels can be systematically integrated during image reconstruction to improve robustness and perceptual fidelity. On our benchmark datasets, our method surpasses the current state-of-the-art methods across most evaluation metrics, with robust gains in perceptual quality metrics.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

PromptFusionSR: Multimodal Enhancement of Low-Resolution Images with Automatic Prompt-Guided Diffusion

  • Chang Qu,
  • Ilhwan Kwon,
  • Karthick Thiyagarajan,
  • Mukesh Prasad,
  • Ali Braytee

摘要

Image super-resolution is a technique aimed at enhancing the resolution of images and is widely utilized in fields such as remote sensing and medical imaging. We view image super-resolution as a reconstruction problem for inferring missing structural information from incomplete observations. This perspective highlights the need for methods that do not merely generate pixel-level corrections, but instead analyze and infer latent geometric and semantic patterns embedded in multimodal data. In this work, we present PromptFusionSR (PF-SR), a novel framework for text-assisted image super-resolution. In contrast to conventional approaches that rely solely on visual features, our method leverages the BLIP3 vision language model to extract contextual prompts-short textual descriptions-directly from low-resolution images. A periodic semantic–structural fusion mechanism enables the model to integrate complementary information channels, enhancing its ability to interpret high-frequency patterns under severe degradation. These textual and information cues guide a pretrained stable diffusion model for 4 \(\times \) upscaling, yielding high-resolution outputs with improved fidelity and richer visual textures. This work further explores how complementary information channels can be systematically integrated during image reconstruction to improve robustness and perceptual fidelity. On our benchmark datasets, our method surpasses the current state-of-the-art methods across most evaluation metrics, with robust gains in perceptual quality metrics.