Recent advancements in neural machine translation (NMT) have opened new possibilities for automatic translation between various language pairs. However, creating high-quality machine translation models remains challenging for many under-resourced languages, such as Kyrgyz and Kazakh. Training of the neural model includes two stages: preprocessing and training itself. This work studies both stages for the Kyrgyz-Kazakh language pair. At the preprocessing stage, text segmentation is important, which seems problematic for morphologically complex languages, including the Kyrgyz and Kazakh languages. Because existing tokenizers can greatly increase the size of the model’s vocabulary, tokenizers based on the relational segmentation model are studied in this work. The second stage, namely, training the neural model of the Kyrgyz-Kazakh language pair, is poorly investigated. Therefore, this work studies a pre-trained T5-small model using existing Kyrgyz-Kazakh language pair datasets in Hugging Face. The scientific contribution of this article is the creation of the morphological tokenizer for Kyrgyz and Kazakh, as well as fine-tuning experiments with this dataset of the neural model T5-small.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Study of Kyrgyz-Kazakh Neural Machine Translation

  • Balzhan Abduali,
  • Ualsher Tukeyev,
  • Zhandos Zhumanov,
  • Nella Israilova

摘要

Recent advancements in neural machine translation (NMT) have opened new possibilities for automatic translation between various language pairs. However, creating high-quality machine translation models remains challenging for many under-resourced languages, such as Kyrgyz and Kazakh. Training of the neural model includes two stages: preprocessing and training itself. This work studies both stages for the Kyrgyz-Kazakh language pair. At the preprocessing stage, text segmentation is important, which seems problematic for morphologically complex languages, including the Kyrgyz and Kazakh languages. Because existing tokenizers can greatly increase the size of the model’s vocabulary, tokenizers based on the relational segmentation model are studied in this work. The second stage, namely, training the neural model of the Kyrgyz-Kazakh language pair, is poorly investigated. Therefore, this work studies a pre-trained T5-small model using existing Kyrgyz-Kazakh language pair datasets in Hugging Face. The scientific contribution of this article is the creation of the morphological tokenizer for Kyrgyz and Kazakh, as well as fine-tuning experiments with this dataset of the neural model T5-small.