<p>Face sketch-photo synthesis (FSPS) plays an important role in applications such as criminal investigation and digital entertainment. Existing deep learning-based FSPS methods are mainly categorized into CNN-based and Transformer-based approaches. CNNs effectively capture local structures but struggle with long-range dependencies, while Transformers excel at global modeling but suffer from quadratic complexity, limiting their practicality in resource-constrained scenarios. Recently, the Mamba selective state space model (SSM) has shown Transformer-level global modeling capability with linear complexity, making it a promising alternative. However, directly combining CNN with vanilla Mamba ignores FSPS’s intrinsic characteristics, thereby limiting the potential of Mamba in this task. To better exploit the strengths of Mamba for FSPS, we propose a Multi-Conditional Prior-guided Dual Spiral CNN-Mamba Network (MCPDS-CMNet), which comprises three core components: a Semantic Prior Dual Spiral CNN-Mamba (SemPDS-CM), a Texture Prior Dual Spiral CNN-Mamba (TexPDS-CM), and a Color-Integrated Feature Fusion (CIFF) module. SemPDS-CM and TexPDS-CM disentangle and enhance semantic and texture features, respectively, enabling more accurate structure reconstruction and improving perceptual realism. In SemPDS-CM and TexPDS-CM, we propose a Dual Spiral Scan (DS-Scan) mechanism to replace conventional scanning methods, aiming to enhance the two models’ capability for modeling long-range dependency in facial structures. CIFF further integrates multi-source features and we introduce an HSV color prior-guided fusion strategy in CIFF to restore color information typically lost during sampling, producing more vivid and natural results. Extensive experiments on four public datasets demonstrate that MCPDS-CMNet achieves competitive performance in synthesis quality and perceptual consistency, validating its effectiveness and excellence.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

A multi-conditional prior-guided dual spiral CNN-Mamba network for face sketch-photo synthesis

  • Yanfei Liu,
  • Miaosen Xu,
  • Yufei Long,
  • Youchang Shi,
  • Hao Wen

摘要

Face sketch-photo synthesis (FSPS) plays an important role in applications such as criminal investigation and digital entertainment. Existing deep learning-based FSPS methods are mainly categorized into CNN-based and Transformer-based approaches. CNNs effectively capture local structures but struggle with long-range dependencies, while Transformers excel at global modeling but suffer from quadratic complexity, limiting their practicality in resource-constrained scenarios. Recently, the Mamba selective state space model (SSM) has shown Transformer-level global modeling capability with linear complexity, making it a promising alternative. However, directly combining CNN with vanilla Mamba ignores FSPS’s intrinsic characteristics, thereby limiting the potential of Mamba in this task. To better exploit the strengths of Mamba for FSPS, we propose a Multi-Conditional Prior-guided Dual Spiral CNN-Mamba Network (MCPDS-CMNet), which comprises three core components: a Semantic Prior Dual Spiral CNN-Mamba (SemPDS-CM), a Texture Prior Dual Spiral CNN-Mamba (TexPDS-CM), and a Color-Integrated Feature Fusion (CIFF) module. SemPDS-CM and TexPDS-CM disentangle and enhance semantic and texture features, respectively, enabling more accurate structure reconstruction and improving perceptual realism. In SemPDS-CM and TexPDS-CM, we propose a Dual Spiral Scan (DS-Scan) mechanism to replace conventional scanning methods, aiming to enhance the two models’ capability for modeling long-range dependency in facial structures. CIFF further integrates multi-source features and we introduce an HSV color prior-guided fusion strategy in CIFF to restore color information typically lost during sampling, producing more vivid and natural results. Extensive experiments on four public datasets demonstrate that MCPDS-CMNet achieves competitive performance in synthesis quality and perceptual consistency, validating its effectiveness and excellence.