VLCA-MFF: enhanced referring image segmentation via visual-linguistic co-attention and multilevel feature fusion
摘要
Referring image segmentation, a fundamental multimodal task, aims to locate all pixels corresponding to the visual object described by a referring expression. Early studies employed visually guided attention at the intermediate layers of visual and verbal encoding stages to highlight target objects; however, they failed to fully leverage the potential of text guidance, leading to insufficient semantic consistency understanding in the models. Additionally, the issue of scale inconsistency among visual features at different levels remained unaddressed, restricting the model performance in complex scenarios. To tackle these challenges, we propose a visual-linguistic co-attention and multilevel feature fusion framework (VLCA-MFF). In the feature extraction stage, this approach integrates BERT with pyramid pooling transformer (P2T) to synchronously capture textual and visual features. A visual-linguistic co-attention module is constructed to consider the bidirectional interaction between text and visual information, aligning text and visual features across multiple encoder levels. This design enhances the semantic consistency of cross-modal information, significantly improves the modeling capability of global contextual information, and deepens the understanding of the overall image scene. Meanwhile, to resolve the multi-scale inconsistency in pyramid feature pooling, this study introduces a multilevel feature fusion module, which integrates the final features of multiple levels during the decoding stage to enhance the spatial consistency of cross-modal information. Experimental results on benchmark datasets such as RefCOCO, RefCOCO+, and G-Ref demonstrate that our method achieves significant improvements.