Image description generation of ethnic minority costumes is an important research direction in the intersection of computer vision and natural language processing, with application potential in cultural protection, education, tourism and other fields. Existing image description methods face challenges such as description simplification, insufficient semantic accuracy and “hallucination phenomenon” in the field of ethnic minority images, and lack of well-annotated datasets. We propose MnCap, a memory-augmented framework designed to generate fine-grained, culturally grounded captions. MnCap consists of three components: (1) a semantic memory bank, constructed via automated triple extraction with expert validation, for cultural knowledge retrieval; (2) a cognitive computing module, which models entities, attributes, and relations through graph-based reasoning; and (3) a multimodal generation module, which aligns visual features with cultural knowledge via attention-based fusion before caption generation. We also introduce MN-28k, a dataset comprising ~ 28,000 images across 55 ethnic groups, with captions initially produced by GPT4o/GLM4V and subsequently refined through three expert validation rounds (annotation consistency: entity-level 0.89, sentence-level 0.82). Experiments demonstrate that MnCap achieves substantial gains in ethnic identification accuracy and cultural symbol coverage while reducing hallucinations. Limitations associated with synthetic annotations are acknowledged, and mitigation strategies are discussed.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

A Vision-Language Fusion Framework for Ethnic Minority Costume Image Captioning

  • Jiahao Zhang,
  • Kuang Hu,
  • Yujing Huang,
  • Dehai Zhang

摘要

Image description generation of ethnic minority costumes is an important research direction in the intersection of computer vision and natural language processing, with application potential in cultural protection, education, tourism and other fields. Existing image description methods face challenges such as description simplification, insufficient semantic accuracy and “hallucination phenomenon” in the field of ethnic minority images, and lack of well-annotated datasets. We propose MnCap, a memory-augmented framework designed to generate fine-grained, culturally grounded captions. MnCap consists of three components: (1) a semantic memory bank, constructed via automated triple extraction with expert validation, for cultural knowledge retrieval; (2) a cognitive computing module, which models entities, attributes, and relations through graph-based reasoning; and (3) a multimodal generation module, which aligns visual features with cultural knowledge via attention-based fusion before caption generation. We also introduce MN-28k, a dataset comprising ~ 28,000 images across 55 ethnic groups, with captions initially produced by GPT4o/GLM4V and subsequently refined through three expert validation rounds (annotation consistency: entity-level 0.89, sentence-level 0.82). Experiments demonstrate that MnCap achieves substantial gains in ethnic identification accuracy and cultural symbol coverage while reducing hallucinations. Limitations associated with synthetic annotations are acknowledged, and mitigation strategies are discussed.