<p>Corpora are crucial in computational linguistics and natural language processing. With the advancement of large language models (LLMs), high-quality human-annotated corpora have become essential for training high-performance general and domain-specific LLMs. This paper presents the construction of a part-of-speech (POS) tagged corpus based on the Twenty-Four Histories and their corresponding modern Chinese translations. First, in the data preprocessing stage, methods such as regular expressions, language models, and manual proofreading were employed to ensure data quality. In addition, task-specific annotation guidelines were established to standardize the POS tagset. Subsequently, the distribution patterns at the lexical level in the constructed corpus were explored from dimensions including word length, word frequency, POS tag distribution, word co-occurrence frequency, POS tag co-occurrence frequency, and word collocation relationships. Finally, we discuss potential applications. This corpus is released to support digital humanities research on ancient Chinese and to facilitate the intelligent processing of classical texts.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Construction of the twenty-four histories ancient-modern part-of-speech tagged corpus

  • Wenhao Ye,
  • Qiankun Xu,
  • Xue Zhao,
  • Dongbo Wang

摘要

Corpora are crucial in computational linguistics and natural language processing. With the advancement of large language models (LLMs), high-quality human-annotated corpora have become essential for training high-performance general and domain-specific LLMs. This paper presents the construction of a part-of-speech (POS) tagged corpus based on the Twenty-Four Histories and their corresponding modern Chinese translations. First, in the data preprocessing stage, methods such as regular expressions, language models, and manual proofreading were employed to ensure data quality. In addition, task-specific annotation guidelines were established to standardize the POS tagset. Subsequently, the distribution patterns at the lexical level in the constructed corpus were explored from dimensions including word length, word frequency, POS tag distribution, word co-occurrence frequency, POS tag co-occurrence frequency, and word collocation relationships. Finally, we discuss potential applications. This corpus is released to support digital humanities research on ancient Chinese and to facilitate the intelligent processing of classical texts.