A Statistical Investigation of Diacritical Ambiguity in Arabic
摘要
Ambiguity is a common phenomenon in natural languages. It takes many forms, including orthographic ambiguity. In Arabic, orthographic ambiguity is manifested by the absence of diacritics in most everyday texts. These diacritics indicate the correct vowelization of words. Hence, we refer to diacritical ambiguity in Arabic. To quantify and analyze diacritical ambiguity in Arabic, we conducted a statistical study at two levels: at the word level with a lexicon of 1.19 million unique diacritized tokens and at the text level with a dataset of 100,000 lines. According to the results, 4 out of 5 letters receive diacritics in Arabic. At the text level, more than 97% of words are ambiguous, meaning they accept more than one possible diacritization, and the average number of diacritization choices is rather high, up to 19.63. These findings demonstrate how significant diacritical ambiguity is in Arabic and how substantial diacritics are in Arabic script.