This paper explores LDA topic modeling using two distinct benchmark datasets: the 20newsgroup dataset and the Reuters21578 dataset. Our main objective is to compare the performance of the LDA model with term TF-IDF scores to baseline models. The study aims to demonstrate how the use of TF-IDF scores enhances the LDA model’s performance in the Reuters dataset to the point where it surpasses the baseline model (from coherence score (Cv) 0.4016–0.5281). However, in the 20newsgroup dataset, the LDA model with TF-IDF scores performs better than the baseline model (from coherence score (Cv) 0.5289–0.5971). These findings demonstrate the value of TF-IDF scoring in enhancing the quality of topic modeling, particularly in diverse textual datasets like newsgroups and news articles. They also stress how important it is to select appropriate text preparation techniques and modify parameters to enhance the performance of LDA models on various text corpora.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Topic Modeling with Latent Dirichlet Allocation (LDA) Using TF-IDF and Bag of Words

  • Tushar Gupta,
  • Arun Kumar Yadav,
  • Mohit Kumar

摘要

This paper explores LDA topic modeling using two distinct benchmark datasets: the 20newsgroup dataset and the Reuters21578 dataset. Our main objective is to compare the performance of the LDA model with term TF-IDF scores to baseline models. The study aims to demonstrate how the use of TF-IDF scores enhances the LDA model’s performance in the Reuters dataset to the point where it surpasses the baseline model (from coherence score (Cv) 0.4016–0.5281). However, in the 20newsgroup dataset, the LDA model with TF-IDF scores performs better than the baseline model (from coherence score (Cv) 0.5289–0.5971). These findings demonstrate the value of TF-IDF scoring in enhancing the quality of topic modeling, particularly in diverse textual datasets like newsgroups and news articles. They also stress how important it is to select appropriate text preparation techniques and modify parameters to enhance the performance of LDA models on various text corpora.