This work presents a modular framework that integrates vision-language models and multimodal Retrieval-Augmented Generation (RAG) to enhance interaction in Serious Games for cultural heritage. The system leverages CLIP to extract visual embeddings from in-game scenes and FAISS to perform efficient similarity search against a pre-indexed image database. Retrieved metadata is used to generate context-aware responses via a locally hosted large language model (LLM), enabling a conversational agent to provide semantically aligned and culturally inclusive dialogue. The architecture consists of three main components: a preprocessing pipeline for image embedding and metadata serialization, a Unity-based client for user interaction, and a FastAPI-based backend for coordinating retrieval and generation. Tested in an archaeological virtual museum, the framework demonstrates improved user engagement, educational potential, and support for adaptive narrative experiences. By combining real-time visual understanding with generative language capabilities, this approach advances the design of interactive and accessible cultural heritage experiences driven by AI.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

A Modular AI-Powered Framework for Semantic Retrieval and Dialogue in Serious Games for Cultural Heritage

  • Simone Pio Barbagallo,
  • Roberto Rizza,
  • Dario Allegra,
  • Anna Maria Gueli,
  • Filippo Stanco

摘要

This work presents a modular framework that integrates vision-language models and multimodal Retrieval-Augmented Generation (RAG) to enhance interaction in Serious Games for cultural heritage. The system leverages CLIP to extract visual embeddings from in-game scenes and FAISS to perform efficient similarity search against a pre-indexed image database. Retrieved metadata is used to generate context-aware responses via a locally hosted large language model (LLM), enabling a conversational agent to provide semantically aligned and culturally inclusive dialogue. The architecture consists of three main components: a preprocessing pipeline for image embedding and metadata serialization, a Unity-based client for user interaction, and a FastAPI-based backend for coordinating retrieval and generation. Tested in an archaeological virtual museum, the framework demonstrates improved user engagement, educational potential, and support for adaptive narrative experiences. By combining real-time visual understanding with generative language capabilities, this approach advances the design of interactive and accessible cultural heritage experiences driven by AI.