Morphological Analysis and Generation › Contextual Analysis in Practice
Language AI HandbookAll chapters Morphological Analysis and Generation › Contextual Analysis in PracticeOne section · Sections in this chapter7 / 12 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Our neural model works on isolated words, which is enough for many analysis tasks. Sentence context becomes important when several analyses survive and must be disambiguated. Many practical pipelines therefore combine contextual part-of-speech or feature prediction with a lemmatization component, but the component order and algorithms differ across toolkits and languages. The code below shows spaCy's user-facing pattern; we mark it eval: false because it requires a separately downloaded model package.
import spacy
nlp = spacy.load("en_core_web_sm")
doc = nlp("The flies were buzzing while she flies to Copenhagen.")
for token in doc:
print(f"{token.text:12s} lemma={token.lemma_:12s} pos={token.pos_}")Running it (after the one-time local install of the model package) shows the two occurrences of flies receive the same lemma but different part-of-speech tags: fly as a plural noun in the first clause, fly as a verb in the second. That distinction comes from sentence context, which an isolated-word analyzer lacks. spaCy's internal pipeline is not the weighted-FST-plus-HMM/CRF architecture from Part VI. Such a combination is one possible design: an FST enumerates lexical analyses, and a contextual sequence model ranks them.