Pan-sharpening fuses low-resolution multispectral (MS) and panchromatic (PAN) images captured synchronously. Deep learning models have demonstrated superior performance in pan-sharpening, as in other domains. More specifically, two-stream architectures have been used for separate feature extraction from the MS and PAN images. Then, a fusion subnetwork followed by an image reconstruction architecture output the MS image with high spatial resolution. However, small details in the output of these pan-sharpening techniques present certain issues. To address this gap, this paper proposes an innovative method that combines traditional and deep learning methodologies. The proposed convolutional neural network (CNN) architecture, named T3IWNet, includes a three-stream encoder for feature extraction from MS and PAN images, along with a discrete wavelet transform (DWT) stream for PAN data decomposition. The fusion module combines and processes MS and PAN features, feeding the resulting tensor to the decoder alongside the DWT stream, PAN, and MS features. The decoder output is the pan-sharpened image. Discarding the DWT stream creates a lower-cost neural network called TIWNet. Computational results are based on a Landsat-8 satellite image dataset. TIWNet and T3IWNet demonstrate outstanding results when compared to traditional and deep learning approaches.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

A Multiscale Encoder-Decoder for Data Fusion in Deep Neural Network for Pan-Sharpening

  • André de Souza Brito,
  • Carla Nascimento Neves,
  • Marcelo Bernardes Vieira,
  • Mauren Louise Sguario Coelho de Andrade,
  • Raul Queiroz Feitosa,
  • José Marcato Junior,
  • Wesley Nunes Gonçalves,
  • Gilson Antonio Giraldi

摘要

Pan-sharpening fuses low-resolution multispectral (MS) and panchromatic (PAN) images captured synchronously. Deep learning models have demonstrated superior performance in pan-sharpening, as in other domains. More specifically, two-stream architectures have been used for separate feature extraction from the MS and PAN images. Then, a fusion subnetwork followed by an image reconstruction architecture output the MS image with high spatial resolution. However, small details in the output of these pan-sharpening techniques present certain issues. To address this gap, this paper proposes an innovative method that combines traditional and deep learning methodologies. The proposed convolutional neural network (CNN) architecture, named T3IWNet, includes a three-stream encoder for feature extraction from MS and PAN images, along with a discrete wavelet transform (DWT) stream for PAN data decomposition. The fusion module combines and processes MS and PAN features, feeding the resulting tensor to the decoder alongside the DWT stream, PAN, and MS features. The decoder output is the pan-sharpened image. Discarding the DWT stream creates a lower-cost neural network called TIWNet. Computational results are based on a Landsat-8 satellite image dataset. TIWNet and T3IWNet demonstrate outstanding results when compared to traditional and deep learning approaches.