Scholarlink

While conventional embedding benchmarks like MTEB provide generalized performance metrics, they fail to adequately evaluate the specialized requirements of embedding models in Retrieval-Augmented Generation (RAG) pipelines. Our IRSC benchmark addresses this gap through targeted assessment of five critical query types (Question, Title, Part-of-Paragraph, Keyword, Summary Retrieval) across multilingual scenarios. For instance, BGE-M3 achieves near-perfect Summary Retrieval (98.12) in IRSC despite moderate MTEB Retrieval scores (54.60), while S-Arctic-L demonstrates paradoxical failure in Title Retrieval (2.48) despite excelling in MTEB Reranking (63.67). These findings validate IRSC’s capacity to expose task-specific competencies masked by conventional metrics. Our key innovation lies in the Similarity of Semantic Comprehension Index (SSCI), which quantifies cross-model semantic alignment through higher-order relationships rather than vector proximity. When applied to 21,000 bilingual queries in CorpusQTPKS, SSCI reveals fundamental differences in how models interpret queries. By bridging the gap between abstract metrics and real-world RAG requirements, IRSC establishes a task-aware evaluation framework that (1) identifies model specialization through multi-scenario testing, (2) enables cross-architecture comparability via SSCI, and (3) provides interpretable diagnostics for retrieval failures. All code, datasets, and a live demo are open-sourced to advance robust RAG system development (Project repository: https://github.com/Jasaxion/IRSC_Benchmark ).

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

IRSC: A Zero-Shot Evaluation Benchmark for Information Retrieval Based on Semantic Comprehension in Retrieval-Augmented Generation Scenarios

  • 会议论文

  • 发表日期：

    2026年

查看全文
  • Hai Lin,
  • Shaoxiong Zhan,
  • Junyou Su,
  • Hai-Tao Zheng,
  • Hui Wang,
  • Xin Su,
  • Ruitong Liu

摘要

While conventional embedding benchmarks like MTEB provide generalized performance metrics, they fail to adequately evaluate the specialized requirements of embedding models in Retrieval-Augmented Generation (RAG) pipelines. Our IRSC benchmark addresses this gap through targeted assessment of five critical query types (Question, Title, Part-of-Paragraph, Keyword, Summary Retrieval) across multilingual scenarios. For instance, BGE-M3 achieves near-perfect Summary Retrieval (98.12) in IRSC despite moderate MTEB Retrieval scores (54.60), while S-Arctic-L demonstrates paradoxical failure in Title Retrieval (2.48) despite excelling in MTEB Reranking (63.67). These findings validate IRSC’s capacity to expose task-specific competencies masked by conventional metrics. Our key innovation lies in the Similarity of Semantic Comprehension Index (SSCI), which quantifies cross-model semantic alignment through higher-order relationships rather than vector proximity. When applied to 21,000 bilingual queries in CorpusQTPKS, SSCI reveals fundamental differences in how models interpret queries. By bridging the gap between abstract metrics and real-world RAG requirements, IRSC establishes a task-aware evaluation framework that (1) identifies model specialization through multi-scenario testing, (2) enables cross-architecture comparability via SSCI, and (3) provides interpretable diagnostics for retrieval failures. All code, datasets, and a live demo are open-sourced to advance robust RAG system development (Project repository: https://github.com/Jasaxion/IRSC_Benchmark ).