Scholarlink

Pairwise text data are essential for pretraining retrieval models. In this work, we propose SDSP (Scalable and Diverse Synthetic Pairwise text generation method), an automatic method to generate scalable and diverse pairwise texts from web corpus data without manual annotation empowered by LLM. First, for text types with limited samples, we build a hierarchical topic creation method to achieve sample set expansion in an iterative way. Second, we propose a multi-task guided diverse data generation scheme to achieve reasonable and controllable high-quality pairwise text generation. Based on an 100M synthetic dataset, an embedding model can be trained to achieve 1.6% and 6.1% performance improvement on MTEB retrieval and STS tasks.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

SDSP: Scalable and Diverse Synthetic Pairwise Text Generation from Web Corpus Using Large Language Model

  • 会议论文

  • 发表日期：

    2026年

查看全文
  • Xiaoxu Wu,
  • Xi Li,
  • Wentao Wu,
  • Aleksei Timofeev,
  • Yinfei Yang,
  • Meng Cao,
  • Ping Huang,
  • Si Li,
  • Jiulong Shan

摘要

Pairwise text data are essential for pretraining retrieval models. In this work, we propose SDSP (Scalable and Diverse Synthetic Pairwise text generation method), an automatic method to generate scalable and diverse pairwise texts from web corpus data without manual annotation empowered by LLM. First, for text types with limited samples, we build a hierarchical topic creation method to achieve sample set expansion in an iterative way. Second, we propose a multi-task guided diverse data generation scheme to achieve reasonable and controllable high-quality pairwise text generation. Based on an 100M synthetic dataset, an embedding model can be trained to achieve 1.6% and 6.1% performance improvement on MTEB retrieval and STS tasks.