Conventional image captioning approaches primarily focus on object region encoding. However, they often overlook contextual dependencies and predicate-level semantics, leading to constrained descriptive precision. To rectify this shortcoming, we introduce HeteroCap, a hierarchical visual-semantic fusion framework designed to enhance scene comprehension and caption generation. Leveraging gated heterogeneous graphs, HeteroCap constructs a multi-modal graph that integrates visual, spatial, and semantic attributes. The aggregation mechanism dynamically filters noise while enabling adaptive cross-modal fusion, constructing a graph that simultaneously encodes spatial layouts and semantic hierarchies. This graph is then refined through hierarchical reasoning layers, which distill object relationships into structured representations. These representations fuel a context-aware Transformer decoder, generating captions that balance linguistic fluency with grounded semantics. Evaluations on MS-COCO and Flickr30k, corroborated by ablation studies, underscore superior performance relative to state-of-the-art models, validating the efficacy of structured knowledge integration in vision-language modeling.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

HeteroCap: Hierarchical Visual-Semantic Fusion with Heterogeneous Graphs for Image Captioning

  • Yuxi Chen,
  • Xiaohua Wu,
  • Zheng Luo,
  • Yuhang Liu

摘要

Conventional image captioning approaches primarily focus on object region encoding. However, they often overlook contextual dependencies and predicate-level semantics, leading to constrained descriptive precision. To rectify this shortcoming, we introduce HeteroCap, a hierarchical visual-semantic fusion framework designed to enhance scene comprehension and caption generation. Leveraging gated heterogeneous graphs, HeteroCap constructs a multi-modal graph that integrates visual, spatial, and semantic attributes. The aggregation mechanism dynamically filters noise while enabling adaptive cross-modal fusion, constructing a graph that simultaneously encodes spatial layouts and semantic hierarchies. This graph is then refined through hierarchical reasoning layers, which distill object relationships into structured representations. These representations fuel a context-aware Transformer decoder, generating captions that balance linguistic fluency with grounded semantics. Evaluations on MS-COCO and Flickr30k, corroborated by ablation studies, underscore superior performance relative to state-of-the-art models, validating the efficacy of structured knowledge integration in vision-language modeling.