Various statistical and neural approaches to Word Sense Disambiguation (WSD) for the Russian language have been extensively studied, yet little attention has been given to leveraging generative LLMs for this task. The ability of generative LLMs to draw insightful conclusions due to their black-box nature and the possibility of interacting with them in natural language make them a promising tool for linguistic research. The aim of this study is to test the ability of generative models, particularly those adapted for Russian language, to distinguish the senses of ambiguous Russian words. The experimental setup we employ involves prompting the model to select the correct sense of an ambiguous word from a predefined list. We conduct the experiment on a range of Russian-language decoder-based LLMs, from state-of-the-art models to smaller ones. The objective of this study is twofold: first, to establish a baseline for generative LLM performance in Russian WSD; second, to conduct a linguistic analysis examining how different models handle various types of lexical ambiguity, namely homonymy and polysemy, as well as their ability to process sparse data. As a result, we outline the performance range across various models, with F1 scores spanning from 0.45 to over 0.9. We also identify linguistic factors influencing disambiguation difficulty, namely the semantic relatedness and representation of the senses. The largest average F1 score of 0.75 is achieved on homonyms with well-represented senses. These findings highlight key areas for future improvement in generative LLMs’ handling of lexical ambiguity in Russian.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Word Sense Disambiguation in Russian: A Generative LLM Approach

  • Polina Gousyatskaya,
  • Natalia Loukachevitch

摘要

Various statistical and neural approaches to Word Sense Disambiguation (WSD) for the Russian language have been extensively studied, yet little attention has been given to leveraging generative LLMs for this task. The ability of generative LLMs to draw insightful conclusions due to their black-box nature and the possibility of interacting with them in natural language make them a promising tool for linguistic research. The aim of this study is to test the ability of generative models, particularly those adapted for Russian language, to distinguish the senses of ambiguous Russian words. The experimental setup we employ involves prompting the model to select the correct sense of an ambiguous word from a predefined list. We conduct the experiment on a range of Russian-language decoder-based LLMs, from state-of-the-art models to smaller ones. The objective of this study is twofold: first, to establish a baseline for generative LLM performance in Russian WSD; second, to conduct a linguistic analysis examining how different models handle various types of lexical ambiguity, namely homonymy and polysemy, as well as their ability to process sparse data. As a result, we outline the performance range across various models, with F1 scores spanning from 0.45 to over 0.9. We also identify linguistic factors influencing disambiguation difficulty, namely the semantic relatedness and representation of the senses. The largest average F1 score of 0.75 is achieved on homonyms with well-represented senses. These findings highlight key areas for future improvement in generative LLMs’ handling of lexical ambiguity in Russian.