Scholarlink

Multilingual language models (MLLMs) like mBERT promise to extend the benefits of NLP research to low-resource languages (LRLs). However, LRL vocabulary is often seriously under-represented in the workpiece dictionaries of MLLMs. This leads to many LRL words being replaced by UNK (unknown tokens) or concatenated from morphologically unrelated wordpieces, consequently leading to low task accuracy. Pre-training MLLMs after including LRL documents is extremely resource-intensive in terms of both human inputs and computational resources. In this chapter, we study intuitive strategies to seek and protect “vulnerable words” in LRLs by introducing them into MLLM dictionaries, providing reasonable initialisations of their embeddings, followed by limited fine-tuning, subject to the limits of available LRL corpora. Our experiments show some significant performance improvements and also some surprising limits to such vocabulary augmentation strategies in various classification tasks in multiple diverse LRLs as well as code-mixed datasets. We release the code and data to enable further research.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Effect of Unknown and Fragmented Tokens on the Performance of Multilingual Language Models at Low-Resource Tasks

  • 章节

  • 出版模式：

    开放获取

  • 发表日期：

    2025年

查看全文
  • Arijit Nag,
  • Bidisha Samanta,
  • Animesh Mukherjee,
  • Niloy Ganguly,
  • Soumen Chakrabarty

摘要

Multilingual language models (MLLMs) like mBERT promise to extend the benefits of NLP research to low-resource languages (LRLs). However, LRL vocabulary is often seriously under-represented in the workpiece dictionaries of MLLMs. This leads to many LRL words being replaced by UNK (unknown tokens) or concatenated from morphologically unrelated wordpieces, consequently leading to low task accuracy. Pre-training MLLMs after including LRL documents is extremely resource-intensive in terms of both human inputs and computational resources. In this chapter, we study intuitive strategies to seek and protect “vulnerable words” in LRLs by introducing them into MLLM dictionaries, providing reasonable initialisations of their embeddings, followed by limited fine-tuning, subject to the limits of available LRL corpora. Our experiments show some significant performance improvements and also some surprising limits to such vocabulary augmentation strategies in various classification tasks in multiple diverse LRLs as well as code-mixed datasets. We release the code and data to enable further research.