This article presents language models, specifically the statistical method known as the N-gram model. Language models are mathematical or computational models that probabilistically represent sequences of words in natural languages (for example, Uzbek, English). They are used to predict the next word based on a given context, generate text, assess word probabilities or analyze language-specific statistical features. The following smoothing methods used in the N-gram model are discussed: Laplace smoothing, Good-Turing discounting, Katz back-off, Interpolation, and Kneser-Ney smoothing. The article presents the stages of building an n-gram language model for Uzbek texts: corpus preparation, N-gram formation, smoothing, language model format preparation, and prediction mechanisms. The corpus frequencies, i.e., the distribution of N-gram counts, are presented both in logarithmic scale and in linear graph form. As a result, when applying Kneser–Ney smoothing for the Uzbek language, the test perplexity of the 3-gram model was 121, for the 5-gram model it was 114, for the 7-gram model it was 112, and for the 9-gram model it was 108.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

N-Gram Language Model for Uzbek Texts

  • Elov Botir Boltayevich,
  • Tojieva Gulbahor Nomozovna,
  • Tokhtaeva Matluba Xakim qizi,
  • Jurayeva Nargiza Jamolidinovna,
  • Primova Mastura Hakim qizi

摘要

This article presents language models, specifically the statistical method known as the N-gram model. Language models are mathematical or computational models that probabilistically represent sequences of words in natural languages (for example, Uzbek, English). They are used to predict the next word based on a given context, generate text, assess word probabilities or analyze language-specific statistical features. The following smoothing methods used in the N-gram model are discussed: Laplace smoothing, Good-Turing discounting, Katz back-off, Interpolation, and Kneser-Ney smoothing. The article presents the stages of building an n-gram language model for Uzbek texts: corpus preparation, N-gram formation, smoothing, language model format preparation, and prediction mechanisms. The corpus frequencies, i.e., the distribution of N-gram counts, are presented both in logarithmic scale and in linear graph form. As a result, when applying Kneser–Ney smoothing for the Uzbek language, the test perplexity of the 3-gram model was 121, for the 5-gram model it was 114, for the 7-gram model it was 112, and for the 9-gram model it was 108.