The automotive sector generates extensive multimodal data, including technical specifications, manuals, and images, yet existing retrieval systems are often limited to unimodal or keyword-based approaches. This paper presents a multimodal Retrieval-Augmented Generation (RAG) system designed for the automobile information domain, where both image and textual data are combined to improve semantic search and question answering. The approach employs OpenAI’s CLIP model for image embeddings and the MiniLM language model for text, integrating them into a dual-indexing pipeline built with FAISS and LlamaIndex for efficient retrieval. User queries are transformed into embeddings that are matched against both text and image indices, and the retrieved context is subsequently provided to a large language model (LLM), Google Gemini, for generating coherent and contextually relevant responses. The system is among the first to apply multimodal RAG techniques on Indian automotive data, achieving Top-1 retrieval accuracy of 82%, Top-3 of 90%, and Top-5 of 94%, with an average query time of 0.32 s. Experimental results indicate that multimodal fusion consistently outperforms unimodal approaches in both accuracy and relevance, providing a strong foundation for intelligent, brand-agnostic automotive assistance systems.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Multi-modal RAG for Text and Image Retrieval

  • M. Nidhi,
  • Dhanya Rao,
  • Saakshi M Vernekar,
  • Pragatilaxmi Itigowni,
  • Uday Kulkarni,
  • D. G. Narayan

摘要

The automotive sector generates extensive multimodal data, including technical specifications, manuals, and images, yet existing retrieval systems are often limited to unimodal or keyword-based approaches. This paper presents a multimodal Retrieval-Augmented Generation (RAG) system designed for the automobile information domain, where both image and textual data are combined to improve semantic search and question answering. The approach employs OpenAI’s CLIP model for image embeddings and the MiniLM language model for text, integrating them into a dual-indexing pipeline built with FAISS and LlamaIndex for efficient retrieval. User queries are transformed into embeddings that are matched against both text and image indices, and the retrieved context is subsequently provided to a large language model (LLM), Google Gemini, for generating coherent and contextually relevant responses. The system is among the first to apply multimodal RAG techniques on Indian automotive data, achieving Top-1 retrieval accuracy of 82%, Top-3 of 90%, and Top-5 of 94%, with an average query time of 0.32 s. Experimental results indicate that multimodal fusion consistently outperforms unimodal approaches in both accuracy and relevance, providing a strong foundation for intelligent, brand-agnostic automotive assistance systems.