Abstract <p>Infrared and visible images provide complementary information that is crucial for complex aerospace perception tasks, yet the scarcity of high-quality infrared aerial datasets and the high cost of data acquisition limit the deployment of advanced computer vision methods. This work proposes CA-Pix2PixHD, a conditional GAN for visible-to-infrared image translation that explicitly models spectral correspondences between the two modalities to synthesize high-fidelity infrared-equivalent images from visible inputs. CA-Pix2PixHD employs a U-Net-based cross-attention multiscale discriminator, which strengthens the representation of thermal textures and improves focus on target regions. The training objective combines adversarial and feature-matching losses with structural similarity loss and gradient-vector loss, enhancing structural consistency as well as edge and contour sharpness. To more rigorously evaluate local generative fidelity, we introduce LSFM, a metric that jointly assesses spatial detail and spectral characteristics. Extensive experiments on three datasets with three state-of-the-art baselines and ablation studies, evaluated by seven quantitative metrics, demonstrate the effectiveness of the proposed method. CA-Pix2PixHD delivers a 20.57<InlineEquation ID="IEq1"> <EquationSource Format="TEX">\(\%\)</EquationSource> </InlineEquation> SSIM improvement on the KAIST dataset, surpasses IRGAN by 35.02<InlineEquation ID="IEq2"> <EquationSource Format="TEX">\(\%\)</EquationSource> </InlineEquation> in <InlineEquation ID="IEq3"> <EquationSource Format="TEX">\(G_T\)</EquationSource> </InlineEquation>, and exceeds Pix2PixHD by 25.33<InlineEquation ID="IEq4"> <EquationSource Format="TEX">\(\%\)</EquationSource> </InlineEquation> in LSFM. These results show that CA-Pix2PixHD generates infrared images with sharper edges and more realistic thermal textures, particularly benefiting airspace target perception and fusion applications.</p> Graphic abstract <p>This figure describes the network architecture of the proposed CA-Pix2PixHD infrared image generation model. The trained model can directly generate a registered Generate IR image from a Real Vis image. During the model training process, a two-stage generator (global and local) is firstly used to generate a Fake IR image that is registered with the Input1 Real Vis image of the aerospace target. In the generator, we integrate the gradient vector loss and the structural similarity loss to improve the overall detail generation and target contour generation effect. Then, the Fake IR image, the Real IR image, and the Semantic image are used as inputs to the discriminator for discrimination to help the generator improve the generation effect. In the discriminator, we use the cross-attention mechanism to dynamically adjust the Fake IR image with the Semantic image as the Query image, so as to focus the discriminant core of the multi-scale discriminator on the texture generation of aerospace targets. Finally, in order to better evaluate the infrared generation effect of aerospace targets, we propose a local feature and frequency matching evaluation index (LSFM). </p>

Ca-pix2pixhd: a cross-attention enhanced conditional GAN for high-quality infrared image synthesis

期刊论文
发表日期：

2026年1月7日

查看全文

Wentao Du,
Kai Shen,
Deqing Huang

摘要

Abstract

Infrared and visible images provide complementary information that is crucial for complex aerospace perception tasks, yet the scarcity of high-quality infrared aerial datasets and the high cost of data acquisition limit the deployment of advanced computer vision methods. This work proposes CA-Pix2PixHD, a conditional GAN for visible-to-infrared image translation that explicitly models spectral correspondences between the two modalities to synthesize high-fidelity infrared-equivalent images from visible inputs. CA-Pix2PixHD employs a U-Net-based cross-attention multiscale discriminator, which strengthens the representation of thermal textures and improves focus on target regions. The training objective combines adversarial and feature-matching losses with structural similarity loss and gradient-vector loss, enhancing structural consistency as well as edge and contour sharpness. To more rigorously evaluate local generative fidelity, we introduce LSFM, a metric that jointly assesses spatial detail and spectral characteristics. Extensive experiments on three datasets with three state-of-the-art baselines and ablation studies, evaluated by seven quantitative metrics, demonstrate the effectiveness of the proposed method. CA-Pix2PixHD delivers a 20.57 \(\%\) SSIM improvement on the KAIST dataset, surpasses IRGAN by 35.02 \(\%\) in \(G_T\) , and exceeds Pix2PixHD by 25.33 \(\%\) in LSFM. These results show that CA-Pix2PixHD generates infrared images with sharper edges and more realistic thermal textures, particularly benefiting airspace target perception and fusion applications.

Graphic abstract

This figure describes the network architecture of the proposed CA-Pix2PixHD infrared image generation model. The trained model can directly generate a registered Generate IR image from a Real Vis image. During the model training process, a two-stage generator (global and local) is firstly used to generate a Fake IR image that is registered with the Input1 Real Vis image of the aerospace target. In the generator, we integrate the gradient vector loss and the structural similarity loss to improve the overall detail generation and target contour generation effect. Then, the Fake IR image, the Real IR image, and the Semantic image are used as inputs to the discriminator for discrimination to help the generator improve the generation effect. In the discriminator, we use the cross-attention mechanism to dynamically adjust the Fake IR image with the Semantic image as the Query image, so as to focus the discriminant core of the multi-scale discriminator on the texture generation of aerospace targets. Finally, in order to better evaluate the infrared generation effect of aerospace targets, we propose a local feature and frequency matching evaluation index (LSFM).