Scholarlink

<p>The digital preservation of Intangible Cultural Heritage (ICH) encounters considerable difficulties due to the limitations of traditional knowledge graphs in representing and integrating multimodal data. In response, a novel Multimodal Alignment (MA) and attention fusion framework is proposed for constructing a comprehensive ICH knowledge graph. Our approach systematically processes textual, image, audio, and video data through a pipeline encompassing Multimodal Fusion (MF), alignment, and joint entity-relation extraction. The core of our method leverages a visual-guided attention mechanism, where features extracted from ICH imagery and keyframes are injected into a Transformer-based text encoder. To deal with the semantic differences of various modalities, an alignment loss technique with Jensen-Shannon Divergence (JSD) is introduced by us. This results in a fully-fledged “relational pattern-ontology mapping-temporal alignment” strategy that is uniquely tailored for knowledge related to ICH. Experimental results on a self-constructed multimodal dataset of Anhui ICH illustrate that our model attains state-of-the-art performance, with an F1-score of 78.95% in triplet extraction, significantly outperforming strong baselines like BLIP and ERNIE-ViL. The constructed knowledge graph, comprising 10,824 nodes, enables intuitive visualization and complex querying, effectively supporting the three-dimensional preservation of ICH practices. This study provides a reusable technical framework and valuable insights for the digital safeguarding and revitalization of ICH.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

A multimodal alignment and attention fusion framework for knowledge graph construction: joint extraction and evaluation in the context of ICH (a case study of Anhui)

  • 期刊论文

  • 发表日期：

    2026年3月24日

查看全文
  • Tiantian Ren,
  • Chao Jiang,
  • Yao Yan

摘要

The digital preservation of Intangible Cultural Heritage (ICH) encounters considerable difficulties due to the limitations of traditional knowledge graphs in representing and integrating multimodal data. In response, a novel Multimodal Alignment (MA) and attention fusion framework is proposed for constructing a comprehensive ICH knowledge graph. Our approach systematically processes textual, image, audio, and video data through a pipeline encompassing Multimodal Fusion (MF), alignment, and joint entity-relation extraction. The core of our method leverages a visual-guided attention mechanism, where features extracted from ICH imagery and keyframes are injected into a Transformer-based text encoder. To deal with the semantic differences of various modalities, an alignment loss technique with Jensen-Shannon Divergence (JSD) is introduced by us. This results in a fully-fledged “relational pattern-ontology mapping-temporal alignment” strategy that is uniquely tailored for knowledge related to ICH. Experimental results on a self-constructed multimodal dataset of Anhui ICH illustrate that our model attains state-of-the-art performance, with an F1-score of 78.95% in triplet extraction, significantly outperforming strong baselines like BLIP and ERNIE-ViL. The constructed knowledge graph, comprising 10,824 nodes, enables intuitive visualization and complex querying, effectively supporting the three-dimensional preservation of ICH practices. This study provides a reusable technical framework and valuable insights for the digital safeguarding and revitalization of ICH.