Multi-modal RAG for Text and Image Retrieval
摘要
The automotive sector generates extensive multimodal data, including technical specifications, manuals, and images, yet existing retrieval systems are often limited to unimodal or keyword-based approaches. This paper presents a multimodal Retrieval-Augmented Generation (RAG) system designed for the automobile information domain, where both image and textual data are combined to improve semantic search and question answering. The approach employs OpenAI’s CLIP model for image embeddings and the MiniLM language model for text, integrating them into a dual-indexing pipeline built with FAISS and LlamaIndex for efficient retrieval. User queries are transformed into embeddings that are matched against both text and image indices, and the retrieved context is subsequently provided to a large language model (LLM), Google Gemini, for generating coherent and contextually relevant responses. The system is among the first to apply multimodal RAG techniques on Indian automotive data, achieving Top-1 retrieval accuracy of 82%, Top-3 of 90%, and Top-5 of 94%, with an average query time of 0.32 s. Experimental results indicate that multimodal fusion consistently outperforms unimodal approaches in both accuracy and relevance, providing a strong foundation for intelligent, brand-agnostic automotive assistance systems.