Enhanced Retrieval-Augmented Generation (RAG) with Question Similarity Filtering for QA Systems
摘要
Quora is a crowdsourced platform for questions and answers. It has a big collection of user-created content. This includes science, technology, medicine, and business. We can use Quora in Retrieval-Augmented Generation (RAG) to enhance large language models (LLMs). We can do it by using its structured question-answer pairs as extra knowledge. To integrate Quora data into an RAG pipeline, we can use transformer-based embedding models. A vector database can store these embeddings. This helps to quickly find answers based on similarity searches. Integrating Quora into RAG brings benefits, but challenges remain. Using Application Programming Interface (API) of services that provide similarity search by LLMs can be very expensive. Also, Quora content often contains biases or low-quality answers. This work uses a hybrid approach and implements the question pair similarity locally without using APIs to reduce the cost of creating the RAG pipeline with LLMs. So, we can implement traditional machine learning models locally and provide strong filters to choose the best questions for LLM. In this work, we review and apply adaptive filtering algorithms to rank Quora responses based on similarity scores and language quality.