Scholarlink

<p>Architectural design relies heavily on rich and multimodal knowledge—including text descriptions, detailed tables, and complex visual information—to inform creative and technical decision-making. However, effectively retrieving and generating meaningful insights from such diverse data sources remains challenging. In this study, we propose Multimodal Retrieval-Augmented Generation (M-RAG), a multimodal retrieval-augmented generation framework that integrates a Multimodal Large Language Model (MLLM) with a unified knowledge base via a shared semantic embedding space, dual-mode table decomposition (text and image), and adaptive query handling to enable traceable cross-modal retrieval and augmented answer generation. We use a multimodal embedding model to project text, table-rendered images, and diagrams into a common space. Quantitative evaluation on the dataset employed in the study reports average text–image similarity of 0.55 ± 0.07; we use an operational threshold <i>θ</i> = 0.5 to flag likely out-of-domain queries and trigger clarification, reducing low-relevance returns. We evaluate the M-RAG framework&#xa0;across multiple architectural scenarios—including reducing technical hallucinations in structural-engineering queries by returning supporting diagrams, enabling special-technical comparison reasoning, handling floorplans and sections, supporting regulation and compliance lookups, retrieving sustainable concept design images and explanations, facilitating material queries, and demonstrating depth-first and breadth-first exploration examples—showing its practical usefulness. By quantifying embedding alignment, retrieval performance, and an operational relevance threshold, M-RAG delivers a more reliable and traceable multimodal retrieval and generation capability for building engineering tasks.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Multimodal retrieval-augmented generation framework for visually rich knowledge in the architecture domain

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2025年11月19日

查看全文
  • Xianchuan Meng,
  • Ziyu Tong

摘要

Architectural design relies heavily on rich and multimodal knowledge—including text descriptions, detailed tables, and complex visual information—to inform creative and technical decision-making. However, effectively retrieving and generating meaningful insights from such diverse data sources remains challenging. In this study, we propose Multimodal Retrieval-Augmented Generation (M-RAG), a multimodal retrieval-augmented generation framework that integrates a Multimodal Large Language Model (MLLM) with a unified knowledge base via a shared semantic embedding space, dual-mode table decomposition (text and image), and adaptive query handling to enable traceable cross-modal retrieval and augmented answer generation. We use a multimodal embedding model to project text, table-rendered images, and diagrams into a common space. Quantitative evaluation on the dataset employed in the study reports average text–image similarity of 0.55 ± 0.07; we use an operational threshold θ = 0.5 to flag likely out-of-domain queries and trigger clarification, reducing low-relevance returns. We evaluate the M-RAG framework across multiple architectural scenarios—including reducing technical hallucinations in structural-engineering queries by returning supporting diagrams, enabling special-technical comparison reasoning, handling floorplans and sections, supporting regulation and compliance lookups, retrieving sustainable concept design images and explanations, facilitating material queries, and demonstrating depth-first and breadth-first exploration examples—showing its practical usefulness. By quantifying embedding alignment, retrieval performance, and an operational relevance threshold, M-RAG delivers a more reliable and traceable multimodal retrieval and generation capability for building engineering tasks.