The language model forms the foundation of NLP. This chapter introduces the N-gram language model and Markov Chains, using the classic literary work The Adventures of Sherlock Holmes by Sir Arthur Conan Doyle (1859–1930) to demonstrate how the N-gram model operates as a cornerstone of NLP in text analysis, followed by the discussion of Shannon’s model, text generation techniques, and evaluation methodologies. This chapter also serves as the conceptual basis for Workshop#2 on N-gram modeling with NLTK in Chap. 12 .

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

N-Gram Language Model

  • Raymond Lee

摘要

The language model forms the foundation of NLP. This chapter introduces the N-gram language model and Markov Chains, using the classic literary work The Adventures of Sherlock Holmes by Sir Arthur Conan Doyle (1859–1930) to demonstrate how the N-gram model operates as a cornerstone of NLP in text analysis, followed by the discussion of Shannon’s model, text generation techniques, and evaluation methodologies. This chapter also serves as the conceptual basis for Workshop#2 on N-gram modeling with NLTK in Chap. 12 .