Scholarlink

<p>TF–IDF is a classical formula that is widely used for identifying important terms within documents. We show that TF–IDF-like scores arise naturally from the test statistic of a penalized likelihood-ratio test setup capturing word burstiness (also known as word over-dispersion). In our framework, the alternative hypothesis captures word burstiness by modeling a collection of documents according to a family of beta-binomial distributions with a gamma penalty term on the precision parameter. In contrast, the null hypothesis assumes that words are binomially distributed in collection documents, a modeling approach that fails to account for word burstiness. We find that a term-weighting scheme given rise to by this test statistic performs comparably to TF–IDF on document classification tasks. This paper provides insights into TF–IDF from a statistical perspective and underscores the potential of hypothesis testing frameworks for advancing term-weighting scheme development.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Common TF–IDF variants arise as key components in the test statistic of a penalized likelihood-ratio test for word burstiness

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年5月18日

查看全文
  • Zeyad Ahmed,
  • Paul Sheridan,
  • Michael McIsaac,
  • Aitazaz A. Farooque

摘要

TF–IDF is a classical formula that is widely used for identifying important terms within documents. We show that TF–IDF-like scores arise naturally from the test statistic of a penalized likelihood-ratio test setup capturing word burstiness (also known as word over-dispersion). In our framework, the alternative hypothesis captures word burstiness by modeling a collection of documents according to a family of beta-binomial distributions with a gamma penalty term on the precision parameter. In contrast, the null hypothesis assumes that words are binomially distributed in collection documents, a modeling approach that fails to account for word burstiness. We find that a term-weighting scheme given rise to by this test statistic performs comparably to TF–IDF on document classification tasks. This paper provides insights into TF–IDF from a statistical perspective and underscores the potential of hypothesis testing frameworks for advancing term-weighting scheme development.