Morphological Analysis and Generation › The Two Directions of Morphological Processing
Language AI HandbookAll chapters Morphological Analysis and Generation › The Two Directions of Morphological ProcessingOne section · Sections in this chapter2 / 12 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Before diving into algorithms, let's be precise about the tasks, because "morphology" in NLP bundles several distinct problems that are easy to conflate.
Analysis (surface to analysis). Given a surface word form, recover one or more candidate analyses. An analysis typically pairs a lemma with a bundle of morphosyntactic features, written here in a Universal Dependencies-style FEATS notation:
- evlerinizden (ev, Number=Plur, Person[psor]=2, Number[psor]=Plur, Case=Abl)
- flies (fly, Number=Plur) or (fly, Person=3, Tense=Pres, VerbForm=Fin)
- watched (watch, Tense=Past, VerbForm=Fin)
Analysis can be ambiguous. The same string may decompose in more than one way, and an analyzer can enumerate those possibilities before a later component selects one. Choosing among them often requires context, which connects directly to the sequence labeling machinery you built in Part VI: hidden Markov models, Viterbi decoding, and conditional random fields. A part-of-speech tagger can help disambiguate whether flies in a particular sentence is the plural noun or the inflected verb.
Generation (analysis to surface). Given a lemma plus features, produce one or more licensed surface strings. This is the direction spell checkers, text-to-speech front ends, and machine translation systems need when they must produce a correctly inflected word in the target language. It is also the direction language models implicitly perform every time they generate an inflected form. Generation is usually less ambiguous than analysis, though a feature bundle can license variants. It is where systematic linguistic knowledge pays off most visibly: get the harmony rule wrong and every affected suffixed form is wrong.
Lemmatization is a frequently used special case of analysis: map a surface form to its dictionary form (its lemma), discarding inflectional detail. It is closely related to the stemming you saw in earlier chapters, but the goals differ in ways that matter for downstream systems.
A stemmer (like the Porter stemmer) chops affixes with string heuristics. It may produce non-words (studies studi) and never distinguishes ambiguity. A lemmatizer targets a dictionary form and may use context: in "The flies were buzzing," flies lemmatizes to fly as a noun, while in "She flies to Rome," it lemmatizes to the same base form as a verb. Stemming is fast and lexicon-free. Lemmatization is often preferable when a downstream task needs linguistically interpretable forms, though its predictions are not error-free and the choice remains task-dependent.
Why not sidestep all of this with the subword tokenizers from Part V? To a first approximation, they do solve a morphology-shaped problem by fragmenting rare words into reusable pieces. BPE greedily merges frequent symbol pairs, while SentencePiece is a toolkit that supports BPE and unigram language-model tokenization. Neither objective requires the resulting units to be linguistically correct morphemes. A BPE vocabulary trained on English may capture the frequent suffix -ing while slicing cancellation into fragments that do not follow its morphological structure. For some language-tokenizer combinations, morphologically rich words split into substantially more tokens, which stretches context windows and changes how much evidence fits into a fixed-length input. We will return to this tension in the limitations section and when we discuss multilingual models in Part XXVI: Multilingual Language Models and Cross-Lingual Transfer.