<p>Existing deep learning-based spectral reconstruction (SR) methods predominantly focus on either global feature extraction or local feature capture, which may result in information loss and thereby compromise the accuracy and robustness of reconstruction. To address this issue, this paper proposes a parallel CNN-Transformer U-Net (PCTUNet) for spectral reconstruction from RGB images. This architecture incorporates dual-branch fusion modules (DBFuses) that integrate a lightweight residual CNN block (LR-CNN) and a dual-attention spectral Transformer block (DAS-TB) in parallel, enabling the complementary extraction of local and global features. Specifically, the DAS-TB branch employs a dual-attention spectral multi-head self-attention (DAMSA) mechanism to capture global features, while the LR-CNN extracts local features. A dedicated interaction–attention fusion module (IAFM), comprising channel attention and spatial attention, is designed within the DBFuse structure to bridge semantic discrepancies between the LR-CNN and DAS-TB pathways, thereby enabling synergistic interaction between global and local information. Experiments conducted on three benchmark datasets (NTIRE2022, CAVE, and Harvard) show that PCTUNet achieves consistent improvements over representative state-of-the-art methods. It reduces MRAE by 6.47–10.29%, lowers RMSE by 8.54–16.57%, increases PSNR by 7.75–9.22%, and decreases SAM by 6.43–17.79%. Moreover, compared with prior approaches, PCTUNet achieves a reduction of up to 50.58% in parameters and 24.20% in computational cost. These results suggest that PCTUNet is a promising and efficient approach for hyperspectral image reconstruction. Moreover, the parallel architecture and significant reduction in computational cost make PCTUNet suitable for deployment in high-performance computing environments, enabling real-time or large-scale hyperspectral image reconstruction. Our code is available at <a href="https://github.com/liuxiy-bo/PCTUNet">https://github.com/liuxiy-bo/PCTUNet</a>.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

PCTUNet: A Parallel CNN-Transformer U-Net for High-Accuracy Spectral Reconstruction from RGB Images

  • Wei Chen,
  • Yi Liu,
  • Jianying Chen,
  • Bin Fan,
  • Zhaohui Li

摘要

Existing deep learning-based spectral reconstruction (SR) methods predominantly focus on either global feature extraction or local feature capture, which may result in information loss and thereby compromise the accuracy and robustness of reconstruction. To address this issue, this paper proposes a parallel CNN-Transformer U-Net (PCTUNet) for spectral reconstruction from RGB images. This architecture incorporates dual-branch fusion modules (DBFuses) that integrate a lightweight residual CNN block (LR-CNN) and a dual-attention spectral Transformer block (DAS-TB) in parallel, enabling the complementary extraction of local and global features. Specifically, the DAS-TB branch employs a dual-attention spectral multi-head self-attention (DAMSA) mechanism to capture global features, while the LR-CNN extracts local features. A dedicated interaction–attention fusion module (IAFM), comprising channel attention and spatial attention, is designed within the DBFuse structure to bridge semantic discrepancies between the LR-CNN and DAS-TB pathways, thereby enabling synergistic interaction between global and local information. Experiments conducted on three benchmark datasets (NTIRE2022, CAVE, and Harvard) show that PCTUNet achieves consistent improvements over representative state-of-the-art methods. It reduces MRAE by 6.47–10.29%, lowers RMSE by 8.54–16.57%, increases PSNR by 7.75–9.22%, and decreases SAM by 6.43–17.79%. Moreover, compared with prior approaches, PCTUNet achieves a reduction of up to 50.58% in parameters and 24.20% in computational cost. These results suggest that PCTUNet is a promising and efficient approach for hyperspectral image reconstruction. Moreover, the parallel architecture and significant reduction in computational cost make PCTUNet suitable for deployment in high-performance computing environments, enabling real-time or large-scale hyperspectral image reconstruction. Our code is available at https://github.com/liuxiy-bo/PCTUNet.