Scholarlink

With over 7,000 languages spoken worldwide and the exponential growth of multilingual digital content, automatically processing multilingual documents has become a crucial challenge. This article presents a comparative study of the state of the art of multilingual text processing systems, describing their architecture, methodology, performance and then their limitations. We systematically valuate large transformer-based models like mBERT, XLM-RoBERTa, mT5, EuroBERT, and KaLM-Embedding sophisticated transformer architectures and conventional Unicode-based segmentation approaches. Results show that mT5 achieves 99.61% accuracy on complex classification tasks, XLM-RoBERTa outperforms in cross-language transfer (+14.6% on XNLI) and low-resource languages ​​(+15.7% in Swahili), EuroBERT leads in long context processing (8,192 tokens), and KaLM-Embedding maximizes efficiency in resource-constrained environments.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

A Comparative Analysis of State-of-the-Art Multilingual Processing Systems

  • 会议论文

  • 发表日期：

    2026年

查看全文
  • Imane Khattabi,
  • Amine Batsi,
  • Samir Boukil,
  • Rachid E. L. Ayachi

摘要

With over 7,000 languages spoken worldwide and the exponential growth of multilingual digital content, automatically processing multilingual documents has become a crucial challenge. This article presents a comparative study of the state of the art of multilingual text processing systems, describing their architecture, methodology, performance and then their limitations. We systematically valuate large transformer-based models like mBERT, XLM-RoBERTa, mT5, EuroBERT, and KaLM-Embedding sophisticated transformer architectures and conventional Unicode-based segmentation approaches. Results show that mT5 achieves 99.61% accuracy on complex classification tasks, XLM-RoBERTa outperforms in cross-language transfer (+14.6% on XNLI) and low-resource languages ​​(+15.7% in Swahili), EuroBERT leads in long context processing (8,192 tokens), and KaLM-Embedding maximizes efficiency in resource-constrained environments.