<p>Stopwords are frequently occurring words in a language with minimal semantic meaning and are commonly removed during text preprocessing to improve the efficiency of natural language processing (NLP) tasks such as sentiment classification and topic modeling. Hindi, despite its widespread use, lacks a readily available, systematically curated stopword list comparable to resource-rich languages such as English. To address this gap, we developed a comprehensive Hindi Stopword List (HiSL) comprising 1311 unique stopwords, constructed through a multi-stage methodology involving translation-based expansion from English and Hindi-English code-mixed text (Hinglish) stopword resources, aggregation of existing Hindi stopword repositories, and corpus-driven statistical extraction using Zipf’s law and knee-point detection across five large-scale, diverse Hindi corpora, followed by manual linguistic validation. We evaluated HiSL on sentiment classification and topic modeling tasks to analyse its practical utility. In sentiment classification, stopword removal using HiSL preserved overall classification accuracy while reducing the token count by 49.13%, indicating substantial preprocessing efficiency without measurable performance loss. In topic modeling experiments using Latent Dirichlet Allocation (LDA) on the BBC Hindi corpus, HiSL produced higher topic coherence (C_v) scores across most topic settings compared with both the no-stopword baseline and the existing LiHiSTO stopword list. Furthermore, HiSL achieved an additional 7% token reduction compared to LiHiSTO while maintaining comparable downstream performance, underscoring its robustness and broader applicability. HiSL is publicly released under an open license to support further research and use in Hindi NLP.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

HiSL: A Comprehensive Hindi Stopword List Derived from Multi-Corpus Statistical and Linguistic Analysis

  • Uday Jaikishor Prasad,
  • Pramod Kumar Mishra

摘要

Stopwords are frequently occurring words in a language with minimal semantic meaning and are commonly removed during text preprocessing to improve the efficiency of natural language processing (NLP) tasks such as sentiment classification and topic modeling. Hindi, despite its widespread use, lacks a readily available, systematically curated stopword list comparable to resource-rich languages such as English. To address this gap, we developed a comprehensive Hindi Stopword List (HiSL) comprising 1311 unique stopwords, constructed through a multi-stage methodology involving translation-based expansion from English and Hindi-English code-mixed text (Hinglish) stopword resources, aggregation of existing Hindi stopword repositories, and corpus-driven statistical extraction using Zipf’s law and knee-point detection across five large-scale, diverse Hindi corpora, followed by manual linguistic validation. We evaluated HiSL on sentiment classification and topic modeling tasks to analyse its practical utility. In sentiment classification, stopword removal using HiSL preserved overall classification accuracy while reducing the token count by 49.13%, indicating substantial preprocessing efficiency without measurable performance loss. In topic modeling experiments using Latent Dirichlet Allocation (LDA) on the BBC Hindi corpus, HiSL produced higher topic coherence (C_v) scores across most topic settings compared with both the no-stopword baseline and the existing LiHiSTO stopword list. Furthermore, HiSL achieved an additional 7% token reduction compared to LiHiSTO while maintaining comparable downstream performance, underscoring its robustness and broader applicability. HiSL is publicly released under an open license to support further research and use in Hindi NLP.