Scholarlink

<p>Weakly supervised semantic segmentation (WSSS), training segmentation models via image-level labels, has the advantage of low manual annotation cost compared with fully supervised semantic segmentation. However, the masks generated by the currently fashionable methods based on the Class Activation Map (CAM) still have the defects of low target segmentation accuracy, many noise pixels and incorrectly activated target pixels. To handle these shortages, the paper proposes a novel WSSS integration network (denoted as WintN-CSG) by adequately fusing the merits of scalability and versatility of multimodal pre-trained basic models <b>C</b>LIP, <b>S</b>AM as well as <b>G</b>rounding-DINO. The superiority of this network with CLIP as a basic framework also benefits from the creative development of Attention Selection Class-aware Attention-based Affinity (AS-CAA), Box Mask Denoising (BMD), and SAM Mask Selection and Fusion (SMSF) modules. Specifically, the proposed AS-CAA can effectively select the representative attention weight maps in Multi-Head Self-Attention (MHSA) to preliminarily remove noise pixels and modify incorrectly activated pixels. Subsequently, the designed BMD combined with Grounding-DINO can shield all noise pixels outside the bounding box, and accurately refine the isolated pixels inside the bounding box, improving the integrity and accuracy of the mask. Furthermore, the deployed SMSF screens out the most suitable masks among many superior mask candidates generated by SAM and makes up for the missing target pixels with the help of fusion and activation algorithms. Finally, experiments with only image-level labels on the PASCAL VOC 2012, MS COCO 2014 and CitySpace datasets show that our scheme achieves excellent performance in the efficiency of mask generation and segmentation accuracy.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

WintN-CSG: a weakly supervised semantic segmentation network based on basic multimodal large-scale pre-trained models

  • 期刊论文

  • 发表日期：

    2025年5月28日

查看全文
  • Haotian Wen,
  • Derui Ding,
  • Wei Liang,
  • Ying Sun

摘要

Weakly supervised semantic segmentation (WSSS), training segmentation models via image-level labels, has the advantage of low manual annotation cost compared with fully supervised semantic segmentation. However, the masks generated by the currently fashionable methods based on the Class Activation Map (CAM) still have the defects of low target segmentation accuracy, many noise pixels and incorrectly activated target pixels. To handle these shortages, the paper proposes a novel WSSS integration network (denoted as WintN-CSG) by adequately fusing the merits of scalability and versatility of multimodal pre-trained basic models CLIP, SAM as well as Grounding-DINO. The superiority of this network with CLIP as a basic framework also benefits from the creative development of Attention Selection Class-aware Attention-based Affinity (AS-CAA), Box Mask Denoising (BMD), and SAM Mask Selection and Fusion (SMSF) modules. Specifically, the proposed AS-CAA can effectively select the representative attention weight maps in Multi-Head Self-Attention (MHSA) to preliminarily remove noise pixels and modify incorrectly activated pixels. Subsequently, the designed BMD combined with Grounding-DINO can shield all noise pixels outside the bounding box, and accurately refine the isolated pixels inside the bounding box, improving the integrity and accuracy of the mask. Furthermore, the deployed SMSF screens out the most suitable masks among many superior mask candidates generated by SAM and makes up for the missing target pixels with the help of fusion and activation algorithms. Finally, experiments with only image-level labels on the PASCAL VOC 2012, MS COCO 2014 and CitySpace datasets show that our scheme achieves excellent performance in the efficiency of mask generation and segmentation accuracy.