Lexical Analysis
摘要
For machines to understand and structure the input text, breaking down text into smaller units and categorizing them is an essential process. Lexical analysis exactly does this; it involves breaking down the input text into its utmost basic units like words, morphemes, and tokens. Morphemes are the smallest units of meaning in a language. They represent distinct words, like “dog” or “read.” Morphemes also includes parts of words, such as prefixes (“un-”), suffixes (“-ing”), or roots (“tele-” as in “telecommunication”). Morphemes are fundamental blocks in the structure of any language.