<p>Referring image segmentation, a fundamental multimodal task, aims to locate all pixels corresponding to the visual object described by a referring expression. Early studies employed visually guided attention at the intermediate layers of visual and verbal encoding stages to highlight target objects; however, they failed to fully leverage the potential of text guidance, leading to insufficient semantic consistency understanding in the models. Additionally, the issue of scale inconsistency among visual features at different levels remained unaddressed, restricting the model performance in complex scenarios. To tackle these challenges, we propose a visual-linguistic co-attention and multilevel feature fusion framework (VLCA-MFF). In the feature extraction stage, this approach integrates BERT with pyramid pooling transformer (P2T) to synchronously capture textual and visual features. A visual-linguistic co-attention module is constructed to consider the bidirectional interaction between text and visual information, aligning text and visual features across multiple encoder levels. This design enhances the semantic consistency of cross-modal information, significantly improves the modeling capability of global contextual information, and deepens the understanding of the overall image scene. Meanwhile, to resolve the multi-scale inconsistency in pyramid feature pooling, this study introduces a multilevel feature fusion module, which integrates the final features of multiple levels during the decoding stage to enhance the spatial consistency of cross-modal information. Experimental results on benchmark datasets such as RefCOCO, RefCOCO+, and G-Ref demonstrate that our method achieves significant improvements.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

VLCA-MFF: enhanced referring image segmentation via visual-linguistic co-attention and multilevel feature fusion

  • Lixia Ji,
  • Yunlong Du,
  • Zhengjie Gong,
  • Ziliang Feng,
  • Han Zhang

摘要

Referring image segmentation, a fundamental multimodal task, aims to locate all pixels corresponding to the visual object described by a referring expression. Early studies employed visually guided attention at the intermediate layers of visual and verbal encoding stages to highlight target objects; however, they failed to fully leverage the potential of text guidance, leading to insufficient semantic consistency understanding in the models. Additionally, the issue of scale inconsistency among visual features at different levels remained unaddressed, restricting the model performance in complex scenarios. To tackle these challenges, we propose a visual-linguistic co-attention and multilevel feature fusion framework (VLCA-MFF). In the feature extraction stage, this approach integrates BERT with pyramid pooling transformer (P2T) to synchronously capture textual and visual features. A visual-linguistic co-attention module is constructed to consider the bidirectional interaction between text and visual information, aligning text and visual features across multiple encoder levels. This design enhances the semantic consistency of cross-modal information, significantly improves the modeling capability of global contextual information, and deepens the understanding of the overall image scene. Meanwhile, to resolve the multi-scale inconsistency in pyramid feature pooling, this study introduces a multilevel feature fusion module, which integrates the final features of multiple levels during the decoding stage to enhance the spatial consistency of cross-modal information. Experimental results on benchmark datasets such as RefCOCO, RefCOCO+, and G-Ref demonstrate that our method achieves significant improvements.