Does Retrieval-Augmented Generation Mitigate Training Data Leakage Risks from Large Language Models?
摘要
Retrieval-augmented generation (RAG) is a technique that enhances training efficiency, knowledge updating, and reliability of large language models (LLMs) by retrieving relevant knowledge from an external knowledge database (retrieval database). While the usefulness of RAG is gaining attention, security and privacy risks associated with RAG systems are beginning to be pointed out. Previous studies have revealed the risk of information leakage related to data within retrieval databases, which is unique to RAG systems, but it has been reported that RAG may mitigate the risk of information leakage related to LLM training data. Although various machine learning systems have been evaluated for the risk of information leakage from trained models through membership inference attacks (MIAs), the evaluation of MIAs on RAG systems has not been sufficiently conducted. In particular, no studies have yet conducted MIAs focusing on the risk of information leakage related to LLM training data on RAG systems, and the potential risk mitigation effects of RAG have not been evaluated. This paper reports the results of the evaluation of the risk mitigation effect of RAG on information leakage related to LLM training data through MIA. Experimental evaluation of RAG systems using three LLMs with 7 billion parameters and two datasets revealed that, contrary to the previous work, RAG does not mitigate the risk of training data leakage from LLMs in MIAs.