Data Leakage Detection in Large Vision-Language Models via Multimodal Perturbation
摘要
The training data for large vision-language models (LVLMs) is typically sourced from large-scale corpora, which may inadvertently include copyrighted or sensitive content, raising concerns about private data leakage. However, detecting such leakage remains challenging due to the opaque internal mechanisms of LVLMs. We observe that memorization in large language models (LLMs) can cause LVLMs to generate distinct responses to seen versus unseen inputs. This discrepancy offers a viable signal for privacy leakage detection. In this paper, we propose DLD-MP, a novel framework for Data Leakage Detection in LVLMs through Multimodal Perturbation. DLD-MP comprises three components: Multi-Level Image Perturbation (MLIP), Key Semantic Mask-based Text Perturbation (KSMTP), and a Leakage Evaluator (LE). Given an image-text pair, MLIP applies perturbations to the image at multiple semantic levels, while KSMTP selectively masks key semantic tokens within the corresponding text. The perturbed inputs are then fed into the target LVLM to perform masked text prediction and vision-language understanding tasks. LE assesses the model’s responses against predefined rules to determine whether the input is used during training. Extensive experiments on multiple benchmarks demonstrate the effectiveness of DLD-MP.