Design and Implementation of a Flexible Translation Engine for Automated Website Crawling and Text Chunking Applied on the Metal Market as an Example
摘要
This study presents the design and implementation of a flexible translation engine tailored for automated website crawling and text chunking, with a specific application to the secondary metal market. Addressing the increasing demand for multilingual processing in Natural Language Processing (NLP), the system integrates Neural Machine Translation (NMT) and Statistical Machine Translation (SMT) with preprocessing techniques such as text chunking to enhance semantic coherence and domain-specific translation accuracy. The study evaluates various chunking and translation strategies using languages relevant to the metal industry, namely Turkish, Chinese, and Spanish, to uncover information often overlooked in English-language sources. Using crawled data from non-English news sites, the system preprocesses, segments, and translates content using models like Multilingual Bidirectional and Auto-Regressive Transformer (mBART) and Large Language Model Meta AI (LLaMA). The translation quality is evaluated using three different metrics, with a special focus on the accuracy of technical terminology. Results show that chunking enhances both translation quality and analysis accuracy by preserving contextual information, while NMT models outperform SMT in domain-specific term handling. The study proposes a transferable framework adaptable to other specialised industries, contributing valuable insights into multilingual data processing and automated market analysis.