Multi-modal content nowadays has become increasingly prevalent in modern applications. However, most existing RAG-based LLMs still focus on textual data, and fail to understand rich semantic information in multi-modal content. To overcome the issue above, we demonstrate a novel Multi-Modal Knowledge Graph (MMKG)-based RAG system, namely MMKG-RAG. After constructing comprehensive multi-modal knowledge graphs, MMKG-RAG provides the foundation for sophisticated multi-modal content retrieval and question answering capabilities. Compared to the multi-modal LLMs with no MMKG, the developed MMKG-RAG offers more powerful understanding and retrieval, and meanwhile improves its reasoning capability across modalities.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

MMKG-RAG: Retrieval-Augmented Generation with Multi-modal Knowledge Graph

  • Shuaitao Zhao,
  • Shijie Luo,
  • Xinyuan Lu,
  • Weixiong Rao

摘要

Multi-modal content nowadays has become increasingly prevalent in modern applications. However, most existing RAG-based LLMs still focus on textual data, and fail to understand rich semantic information in multi-modal content. To overcome the issue above, we demonstrate a novel Multi-Modal Knowledge Graph (MMKG)-based RAG system, namely MMKG-RAG. After constructing comprehensive multi-modal knowledge graphs, MMKG-RAG provides the foundation for sophisticated multi-modal content retrieval and question answering capabilities. Compared to the multi-modal LLMs with no MMKG, the developed MMKG-RAG offers more powerful understanding and retrieval, and meanwhile improves its reasoning capability across modalities.