The traditional NMT model based on word or subword tokenization constitutes the prevalent technique; however, it encounters the rigidity of language-specific tokenizers, the OOV dilemma, and representation constraints, which are particularly evident in multilingual settings, leading to the over-segmentation of low-resource languages and subsequent degradation of translation performance. To address these issues, this paper introduces a byte-based tokenization scheme, which employs UTF-8 encoding to transform input sequences into byte sequences and aggregates byte information through a multi-scale convolutional layer to enhance the learning of character/word-level representations. Experimental results indicate that, compared to traditional byte-based models, this approach yields a significant performance enhancement in many-to-one translation tasks, with an average BLEU score increase of 1.9, effectively mitigating representation bottlenecks and demonstrating potential in low-resource language translation.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

From Bytes to Words: Enhancing Multilingual NMT Through Explicit Byte Integration

  • Sen Lin,
  • Turdi Tohti,
  • Yi Liang,
  • Askar Hamdulla

摘要

The traditional NMT model based on word or subword tokenization constitutes the prevalent technique; however, it encounters the rigidity of language-specific tokenizers, the OOV dilemma, and representation constraints, which are particularly evident in multilingual settings, leading to the over-segmentation of low-resource languages and subsequent degradation of translation performance. To address these issues, this paper introduces a byte-based tokenization scheme, which employs UTF-8 encoding to transform input sequences into byte sequences and aggregates byte information through a multi-scale convolutional layer to enhance the learning of character/word-level representations. Experimental results indicate that, compared to traditional byte-based models, this approach yields a significant performance enhancement in many-to-one translation tasks, with an average BLEU score increase of 1.9, effectively mitigating representation bottlenecks and demonstrating potential in low-resource language translation.