Construction of multimodal knowledge graph for cultural heritage: enhanced joint extraction and domain-adaptive cross-modal alignment method
摘要
The structured organization and semantic understanding of multimodal cultural heritage data is a key research direction in the field of digital humanities. However, when processing cultural heritage text and image data, existing methods still face key challenges including limited accuracy of entity and relation extraction, insufficient modeling ability for overlapping triples, and difficulties in cross-modal semantic alignment. To address the above challenges, this paper proposes a cultural heritage-oriented construction method for multimodal knowledge graphs. First, a domain-standardized multimodal dataset is constructed, which provides a data foundation for model training and evaluation. For entity and relation extraction, an enhanced joint extraction model BS-PTWF is proposed. Through the textual temporal feature enhancement strategy, the relative position-aware task-specific multi-head attention mechanism, and the dual-path subject representation method, it effectively improves the extraction performance under complex semantic conditions, especially for overlapping triples. It achieves an F1-Score of 82.5% on the self-constructed dataset, which is 4.8 percentage points higher than the baseline model and 2.9 percentage points higher than the state-of-the-art comparative model, and reaches F1-Scores of 61.0% and 80.6% on the CMeIE and DuIE2.0 datasets respectively. For cross-modal alignment, domain-adaptive fine-tuning is performed on the Chinese-adapted CLIP pre-trained model, which realizes fine-grained semantic alignment of cultural heritage images and texts, with a maximum Recall@10 (R@10) of 97.4% in the image-to-text retrieval task. On this basis, a multimodal knowledge graph is constructed by fusing text and image modality information, and a visual interactive system is designed to support knowledge query and multi-dimensional display.