TL; DR: Text Normalization for Social Media Corpus
摘要
Text annotation is the process of enriching text with metalinguistic information. The most common types of annotation include morphological annotation, which involves assigning grammatical tags to words; syntactic annotation, which identifies syntactic relationships within sentences; and lemmatization, which determines the base form (lemma) of each word. Text annotation is a fundamental task in computational linguistics and plays a crucial role in both linguistic research and practical applications of natural language processing (NLP). Given the scale of modern text corpora (exceeding 1 billion words), manual annotation is virtually unfeasible, making automated annotation tools the only viable solution. However, most of these tools have been developed, trained, and tested on texts with standard orthography. As a result, their performance can significantly degrade when applied to non-standard texts, such as those from social media. This issue can be mitigated through automated text normalization prior to annotation. This task is related to automated spelling correction but is not equivalent to it, and remains significantly less studied. For instance, text normalization requires not only correcting mistyping and spelling errors but also restoring abbreviations commonly used in online communication. To explore approaches to text normalization, we compiled a corpus of social media sentences and manually paired them with their normalized versions. Using this corpus, we compared various text normalization methods, leveraging pre-trained language models. We examined both fine-tuning and prompt-based approaches. Our study has identified the most effective strategies in this domain, providing valuable insights for future research and applications.