The widespread digitization of cultural heritage (CH) collections has enabled institutions to make vast artefacts accessible through knowledge graphs (KGs). Yet existing retrieval systems predominantly rely on keyword-based search with semantic enrichment via linking to controlled vocabularies, while only a few systems support visual information search, and none support queries involving relational metadata constraints such as temporal ranges, or combinations of metadata and visual information. Although advanced approaches, including multimodal retrieval and knowledge reasoning, have emerged in research, few have been deployed in operational CH environments, creating a substantial gap between academic innovations and real-world practice. In this paper, we present a knowledge-enhanced multimodal retrieval system that operates over KG, integrating a domain-adaptive fine-tuned CLIP model with an LLM-based Text2SPARQL and unifying their results through a fusion strategy. We deployed the system as a backend API integrated into a CH platform and conducted a comprehensive evaluation through quantitative metrics and qualitative user studies with museum curators. Results demonstrate that our domain-adaptive CLIP achieves high retrieval accuracy, while Text2SPARQL effectively further enhances precision through knowledge reasoning. Our work provides practical experience into the benefits and limitations of these technologies in production environments, informing broader adoption across CH institutions.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Knowledge-Enhanced Multimodal Retrieval over Cultural Heritage Knowledge Graphs

  • Xuemin Duan,
  • Federico D’Asaro,
  • Ruben Peeters,
  • Giacomo Blanco,
  • Tommaso Monopoli,
  • Giuseppe Rizzo,
  • Anastasia Dimou

摘要

The widespread digitization of cultural heritage (CH) collections has enabled institutions to make vast artefacts accessible through knowledge graphs (KGs). Yet existing retrieval systems predominantly rely on keyword-based search with semantic enrichment via linking to controlled vocabularies, while only a few systems support visual information search, and none support queries involving relational metadata constraints such as temporal ranges, or combinations of metadata and visual information. Although advanced approaches, including multimodal retrieval and knowledge reasoning, have emerged in research, few have been deployed in operational CH environments, creating a substantial gap between academic innovations and real-world practice. In this paper, we present a knowledge-enhanced multimodal retrieval system that operates over KG, integrating a domain-adaptive fine-tuned CLIP model with an LLM-based Text2SPARQL and unifying their results through a fusion strategy. We deployed the system as a backend API integrated into a CH platform and conducted a comprehensive evaluation through quantitative metrics and qualitative user studies with museum curators. Results demonstrate that our domain-adaptive CLIP achieves high retrieval accuracy, while Text2SPARQL effectively further enhances precision through knowledge reasoning. Our work provides practical experience into the benefits and limitations of these technologies in production environments, informing broader adoption across CH institutions.