<p>Image captioning is a cross-modal task that combines computer vision and natural language processing to generate natural language descriptions of visual content. Recent advances have explored the integration of both grid-based and region-based visual features to better capture relational and contextual information, such as interactions between objects and their surrounding environment. However, combining multiple types of features often results in spatial misalignment, which hinders the model’s ability to construct coherent visual-semantic relationships. To address these limitations, we propose a novel Spatial Alignment Positional Encoder(SAPE), which encodes spatial information across aligned grid-level and region-level features to construct a unified visual-spatial representation. In addition, we introduce two complementary enhancement modules: Group Normalization Multi-head Attention(GNMA) to capture global relational cues, and Convolution-based Feature Enhancement Attention(CFEA) to enrich local spatial details. To mitigate the degradation of positional signals during deep training, we further propose a Learnable Adaptive Positional Encoder(LAPE) that dynamically preserves position-sensitive information. These components are integrated into a unified transformer-based architecture named the Visual-Spatial Relationship Sensitive Transformer(VRST). Extensive experiments on the MSCOCO dataset demonstrate the effectiveness of our approach, achieving a CIDEr score of 141.9 on the Karpathy test split and 138.2 on the official evaluation server, surpassing several strong baselines.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Visual spatial relationship sensitive transformer for image captioning

  • Xianghua Piao,
  • Dong Jin,
  • Min Jung Kwon,
  • Yeong Hyeon Gu

摘要

Image captioning is a cross-modal task that combines computer vision and natural language processing to generate natural language descriptions of visual content. Recent advances have explored the integration of both grid-based and region-based visual features to better capture relational and contextual information, such as interactions between objects and their surrounding environment. However, combining multiple types of features often results in spatial misalignment, which hinders the model’s ability to construct coherent visual-semantic relationships. To address these limitations, we propose a novel Spatial Alignment Positional Encoder(SAPE), which encodes spatial information across aligned grid-level and region-level features to construct a unified visual-spatial representation. In addition, we introduce two complementary enhancement modules: Group Normalization Multi-head Attention(GNMA) to capture global relational cues, and Convolution-based Feature Enhancement Attention(CFEA) to enrich local spatial details. To mitigate the degradation of positional signals during deep training, we further propose a Learnable Adaptive Positional Encoder(LAPE) that dynamically preserves position-sensitive information. These components are integrated into a unified transformer-based architecture named the Visual-Spatial Relationship Sensitive Transformer(VRST). Extensive experiments on the MSCOCO dataset demonstrate the effectiveness of our approach, achieving a CIDEr score of 141.9 on the Karpathy test split and 138.2 on the official evaluation server, surpassing several strong baselines.