This paper presents ASSIST-AI, a novel multi-modal multi-agentic framework designed to enhance human-computer interaction in cultural heritage environments through advanced AI technologies. Our system integrates computer vision, natural language processing, and speech technologies to create context-aware conversational agents for museum settings. The framework combines Retrieval-Augmented Generation (RAG) with automatic Point of Interest (PoI) detection, personalized user profiling, and real-time multi-modal interaction capabilities. We demonstrate significant improvements in user engagement through adaptive personalization mechanisms that leverage attention schema theory and contextual awareness. Evaluation with 30 participants across major Spanish museums shows enhanced visitor experience and knowledge retention. The system achieves high accuracy in artwork identification, sub-second response times for multi-modal queries, and supports real-time interaction in Spanish and English with adaptive complexity based on user expertise levels.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

ASSIST: A Multi-agentic Framework for Human Computer Interaction in Cultural Heritage Settings

  • Samuel Ramos-Varela,
  • Anmol Guragain,
  • Jaime Bellver-Soler,
  • David Aragon Diaz,
  • Long Lin,
  • Luis Fernando D’Haro

摘要

This paper presents ASSIST-AI, a novel multi-modal multi-agentic framework designed to enhance human-computer interaction in cultural heritage environments through advanced AI technologies. Our system integrates computer vision, natural language processing, and speech technologies to create context-aware conversational agents for museum settings. The framework combines Retrieval-Augmented Generation (RAG) with automatic Point of Interest (PoI) detection, personalized user profiling, and real-time multi-modal interaction capabilities. We demonstrate significant improvements in user engagement through adaptive personalization mechanisms that leverage attention schema theory and contextual awareness. Evaluation with 30 participants across major Spanish museums shows enhanced visitor experience and knowledge retention. The system achieves high accuracy in artwork identification, sub-second response times for multi-modal queries, and supports real-time interaction in Spanish and English with adaptive complexity based on user expertise levels.