Image Inpainting refers to the process in computer vision task in which missing parts in a region is completed. Early methods were based on patch propagation in which the surrounding pixels were used to complete the image with missing regions. Such techniques were limited in handling complex information. Advancements were done in the deep neural networks through Convolutional Neural Networks (CNNs) and Generative Adversarial Networks (GANs) which significantly improved the inpainting process. But these methods suffered from instability. The most recent advancements are the transformer architecture and diffusion models. These have established very high benchmark values. These models are able to capture long range dependencies. As a result, these can produce very high-quality images. This paper presents a comparative study of these methods. It tracks the evolution of methods from classical algorithm to generative AI (Artificial Intelligence). The study focuses on some popular datasets – Places2, CelebA-HQ and also ParisStreetView. Using these datasets, we evaluate the performance of six models. The evaluation metrics consists of pixel and structure level metrics such as PSNR (Peak Signal-to-Noise Ratio) and SSIM (Structural Similarity Index Measure). There are other high level metrics such as LPIPS (Learned Perceptual Image Patch Similarity) and FID (Frechet Inception Distance) values. Using the evaluation metric, the quantitative performance of the models is analyzed. Further, the qualitative performance of the models is evaluated. This is done by conducting a user study. The qualitative and quantitative evaluation allows us to perform a comprehensive comparative analysis. The results show that diffusion-based models achieve the highest image quality, while the transformer architecture achieves highest consistency. The study further identifies the challenges and future direction in image inpainting.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Advances in Image Inpainting: A Comparative Study of CNN, GAN, Transformer, and Diffusion-Based Models

  • Megha Upreti,
  • Ravindra Singh Koranga,
  • Shashi Kumar Sharma,
  • Vaishali Dev

摘要

Image Inpainting refers to the process in computer vision task in which missing parts in a region is completed. Early methods were based on patch propagation in which the surrounding pixels were used to complete the image with missing regions. Such techniques were limited in handling complex information. Advancements were done in the deep neural networks through Convolutional Neural Networks (CNNs) and Generative Adversarial Networks (GANs) which significantly improved the inpainting process. But these methods suffered from instability. The most recent advancements are the transformer architecture and diffusion models. These have established very high benchmark values. These models are able to capture long range dependencies. As a result, these can produce very high-quality images. This paper presents a comparative study of these methods. It tracks the evolution of methods from classical algorithm to generative AI (Artificial Intelligence). The study focuses on some popular datasets – Places2, CelebA-HQ and also ParisStreetView. Using these datasets, we evaluate the performance of six models. The evaluation metrics consists of pixel and structure level metrics such as PSNR (Peak Signal-to-Noise Ratio) and SSIM (Structural Similarity Index Measure). There are other high level metrics such as LPIPS (Learned Perceptual Image Patch Similarity) and FID (Frechet Inception Distance) values. Using the evaluation metric, the quantitative performance of the models is analyzed. Further, the qualitative performance of the models is evaluated. This is done by conducting a user study. The qualitative and quantitative evaluation allows us to perform a comprehensive comparative analysis. The results show that diffusion-based models achieve the highest image quality, while the transformer architecture achieves highest consistency. The study further identifies the challenges and future direction in image inpainting.