Scholarlink

Existing multi-modal large language models (MLLMs) exhibit limitations in fine-grained object attribute recognition, spatial-relational modeling, and complex multi-step reasoning, particularly for knowledge-intensive visual question answering (VQA) tasks requiring precise visual grounding. We propose RAG-VA, a retrieval-augmented framework that enhances MLLMs through scene graph-based knowledge representation and a visual element vector database for dynamic information retrieval. RAG-VA implements a multi-stage visual reasoning pipeline comprising visual element observation, information extraction, reasoning chain verification, and guided inference, along with a zero-shot chain-of-thought (CoT) approach for stepwise reasoning without fine-tuning. Extensive evaluations across multiple standard benchmarks demonstrate RAG-VA’s superior performance over advanced MLLMs like LLaVA and ShareGPT4V, particularly in fine-grained visual understanding and knowledge-based reasoning tasks, offering an effective solution for integrating visual and external knowledge in VQA.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

RAG with Visual Alert: Boosting Multimodal Language Models for Enhanced Visual Question Answering

  • 会议论文

  • 发表日期：

    2026年

查看全文
  • Hongze Ou,
  • Xiaoyu Liang,
  • Lianrui Mu,
  • Haoji Hu

摘要

Existing multi-modal large language models (MLLMs) exhibit limitations in fine-grained object attribute recognition, spatial-relational modeling, and complex multi-step reasoning, particularly for knowledge-intensive visual question answering (VQA) tasks requiring precise visual grounding. We propose RAG-VA, a retrieval-augmented framework that enhances MLLMs through scene graph-based knowledge representation and a visual element vector database for dynamic information retrieval. RAG-VA implements a multi-stage visual reasoning pipeline comprising visual element observation, information extraction, reasoning chain verification, and guided inference, along with a zero-shot chain-of-thought (CoT) approach for stepwise reasoning without fine-tuning. Extensive evaluations across multiple standard benchmarks demonstrate RAG-VA’s superior performance over advanced MLLMs like LLaVA and ShareGPT4V, particularly in fine-grained visual understanding and knowledge-based reasoning tasks, offering an effective solution for integrating visual and external knowledge in VQA.