From Bytes to Words: Enhancing Multilingual NMT Through Explicit Byte Integration
摘要
The traditional NMT model based on word or subword tokenization constitutes the prevalent technique; however, it encounters the rigidity of language-specific tokenizers, the OOV dilemma, and representation constraints, which are particularly evident in multilingual settings, leading to the over-segmentation of low-resource languages and subsequent degradation of translation performance. To address these issues, this paper introduces a byte-based tokenization scheme, which employs UTF-8 encoding to transform input sequences into byte sequences and aggregates byte information through a multi-scale convolutional layer to enhance the learning of character/word-level representations. Experimental results indicate that, compared to traditional byte-based models, this approach yields a significant performance enhancement in many-to-one translation tasks, with an average BLEU score increase of 1.9, effectively mitigating representation bottlenecks and demonstrating potential in low-resource language translation.