HeteroCap: Hierarchical Visual-Semantic Fusion with Heterogeneous Graphs for Image Captioning
摘要
Conventional image captioning approaches primarily focus on object region encoding. However, they often overlook contextual dependencies and predicate-level semantics, leading to constrained descriptive precision. To rectify this shortcoming, we introduce HeteroCap, a hierarchical visual-semantic fusion framework designed to enhance scene comprehension and caption generation. Leveraging gated heterogeneous graphs, HeteroCap constructs a multi-modal graph that integrates visual, spatial, and semantic attributes. The aggregation mechanism dynamically filters noise while enabling adaptive cross-modal fusion, constructing a graph that simultaneously encodes spatial layouts and semantic hierarchies. This graph is then refined through hierarchical reasoning layers, which distill object relationships into structured representations. These representations fuel a context-aware Transformer decoder, generating captions that balance linguistic fluency with grounded semantics. Evaluations on MS-COCO and Flickr30k, corroborated by ablation studies, underscore superior performance relative to state-of-the-art models, validating the efficacy of structured knowledge integration in vision-language modeling.