Transformer embedded X-shaped encoding-decoding GAN for NIR-VIS face synthesis
摘要
Current cross-modal face synthesis methods predominantly focus on inter-domain style transfer while neglecting facial identity features, leading to degraded identity preservation. Furthermore, most methods fail to fully utilize style information, resulting in poor style transfer performance under low-light conditions. This paper proposes an example-based Transformer-embedded X-shaped encoding-decoding generative adversarial network that simultaneously handles style, content, and identity feature transformations. We designed a dual-branch encoder with embedded Transformer to simultaneously extract identity and content features, enhancing identity recognition. An attention mechanism is employed in the dual-branch decoder, where focusing on key facial features by masks to improve image resolution. Additionally, a local style attention fusion module is constructed to collaborate with traditional global style fusion modules. This module collaborates with a traditional global style fusion module to extract style information from reference photos, providing rich prior guidance for generating realistic and natural spectral outputs. Optimized by a carefully designed identity preservation loss function, the proposed architecture demonstrates outstanding performance. Extensive experiments conducted on two benchmark datasets demonstrate the effectiveness of the proposed model for both face recognition and high-quality image generation, highlighting its superiority over existing approaches.