Workshop#2: N-Grams Modeling with Natural Language Toolkit (Hour 3–4)
摘要
This workshop, aligned with Chap. 2 , focuses on N-gram generation and statistics using NLTK technology. It is divided into two parts: Part I introduces the N-gram language model using NLTK in Python and the N-grams to generate N-gram statistics from any sentence, text object, or entire document. This provides foundational techniques for text analysis, parsing, and semantic analysis, which will be further explored in subsequent workshops; Part II introduces spaCy, another key NLP Python library, which, like NLTK, is not only valuable for teaching and learning but is also widely used in real-world NLP applications such as text summarization, information extraction, and question-answering chatbots. SpaCy serves as a critical tool that integrates with Transformer technology in later workshops.