Optimizing Text Data Storage for Improved Natural Language Processing: A Comprehensive Analysis
摘要
This research analyzed the qualitative characteristics of messages in the Telegram messenger, which are used as input data for further analysis of text content. The parameters of these messages, such as their format, size, presence of noise, velocity, etc., have been carefully considered. The main goal of the article is to identify the optimal approach to saving a large amount of data before the important stage of text analysis. In the course of the study, a detailed analysis of literature sources devoted to this topic was carried out. The article examines the main advantages and disadvantages of existing data preprocessing algorithms, as well as problems related to data purity and their impact on potential research results. As part of software experiments, the influence of data preprocessing on the saved data size for further use, as well as the input data velocity, was evaluated. Among the proposed methods, the method of saving cleared tokens in string format and saving word codes in string format together with the word-code dictionary was highlighted. This is aimed at ensuring the effective distribution of tasks of the text analysis system during the day.