Scholarlink

Performance of large language models (LLMs) is correlated with diverse and high-quality training data. One aspect of data quality is the presence of duplicate documents in the training data. In this paper, we evaluate five algorithms for deduplicating large data sets, namely MinHash/LSH, Exact Hashes, SimHash, Scalable Bloom filter, and Suffix Array. We report on their precision, recall, memory requirements and runtime on deduplicating OpenSubtitles and Oscar data for five languages (EN, DE, ES, FR, IT). We find that the best overall performance is achieved by using a MinHash/LSH, but other options such as Scalable Bloom filter  can be more suitable in resource-critical situations. While precision varies between 0.833 and 0.985 across algorithms, recall varies between 0.247 and 0.989, indicating different levels of aggressiveness. We conclude that MinHash/LSH is the most suitable algorithm to deduplicate pretraining data for LLMs.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Evaluation of Document Deduplication Algorithms for Large Text Corpora

  • 会议论文

  • 发表日期：

    2025年

查看全文
  • Johannes Leveling,
  • Lennard Helmer,
  • Benny Joerg Stein,
  • Dennis Wegener,
  • Zoha Sheikh,
  • Elanton Fernandes,
  • Hammam Abdelwahab

摘要

Performance of large language models (LLMs) is correlated with diverse and high-quality training data. One aspect of data quality is the presence of duplicate documents in the training data. In this paper, we evaluate five algorithms for deduplicating large data sets, namely MinHash/LSH, Exact Hashes, SimHash, Scalable Bloom filter, and Suffix Array. We report on their precision, recall, memory requirements and runtime on deduplicating OpenSubtitles and Oscar data for five languages (EN, DE, ES, FR, IT). We find that the best overall performance is achieved by using a MinHash/LSH, but other options such as Scalable Bloom filter  can be more suitable in resource-critical situations. While precision varies between 0.833 and 0.985 across algorithms, recall varies between 0.247 and 0.989, indicating different levels of aggressiveness. We conclude that MinHash/LSH is the most suitable algorithm to deduplicate pretraining data for LLMs.