The advancement of large-scale language models in recent years has significantly enhanced various natural language processing (NLP) domains. This research addresses the specific challenge of developing BERT-based models tailored for domain-specific language modeling. Tokenization efficiency within BERT-based models is directly related to model efficiency which in turn motivated the research we present in this paper. This research aims to enhance the development of a domain-adapted large language model for the case of Serbian language. Our focus was the influence of training dataset content on the quality of the domain-adapted large language model tokenizer developed for masked language modelling task. In this paper, we will present a comparison of the tokenization performance for two tokenizers developed using different mixtures of data within two versions of a large language model named SrBERTa.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Domain Adaptation of Large Language Model Tokenizers—A Comparison of Performances

  • Miloš Bogdanović,
  • Milena Frtunić Gligorijević,
  • Jelena Kocić,
  • Leonid Stoimenov

摘要

The advancement of large-scale language models in recent years has significantly enhanced various natural language processing (NLP) domains. This research addresses the specific challenge of developing BERT-based models tailored for domain-specific language modeling. Tokenization efficiency within BERT-based models is directly related to model efficiency which in turn motivated the research we present in this paper. This research aims to enhance the development of a domain-adapted large language model for the case of Serbian language. Our focus was the influence of training dataset content on the quality of the domain-adapted large language model tokenizer developed for masked language modelling task. In this paper, we will present a comparison of the tokenization performance for two tokenizers developed using different mixtures of data within two versions of a large language model named SrBERTa.