Morphology and Word Formation › Why Word Structure Matters for Language Models

It is worth pausing to trace exactly where morphology enters the pipelines you have already built. None of the following points requires new machinery; each is a familiar failure mode whose root cause turns out to be linguistic.

Statistical sparsity. N-gram language models estimate probabilities by counting. Inflection multiplies events: instead of learning one distribution over walk, you learn separate distributions over walk, walks, walked, and walking, and your counts for each thin dramatically. Smoothing techniques from Part II mitigate this, but they are patches over a structural problem: the model has no way to know that walked and strolled behave similarly because both are past-tense forms. Smoothing redistributes probability mass according to generic assumptions; morphology would redistribute it according to the actual generative structure of the data.

Out-of-vocabulary rates. Any word-form-level vocabulary misses rare inflections, novel derivations, and fresh compounds. Morphology explains both why these forms exist (productivity) and what they mean given their parts, which is precisely the knowledge a subword tokenizer approximates statistically. When you encounter an unseen form like unputdownable, morphology tells you it is comprehensible in principle; a closed vocabulary simply reports a miss.

Embedding coverage. As we saw in the FastText chapter of Part IV, representing words as bags of character n-grams lets rare words borrow strength from their substrings: superstitious shares n-grams with superstition. FastText works precisely because character n-grams approximate morphemes. When the approximation aligns with true morphological structure, embeddings generalize well; when it does not, they inherit noise from unrelated words. The n-gram bag has no way to prefer super- over coincidental overlaps, so its generalizations are only as good as the accidental alignment between substrings and meaning.

Subwords as implicit morphology. BPE, WordPiece, and Unigram tokenization learn frequent substrings, and in practice many learned tokens correspond to real morphemes: common prefixes, suffixes, and stems survive as units. But the correspondence is loose. Frequency-driven merging often preserves frequent strings across word boundaries (as with there inside therefore), fragments rare affixes inconsistently across similar words, and carves words like Wednesday into chunks bearing no semantic relation to their parts. Understanding morphology lets you diagnose these failures and evaluate whether a tokenizer's segmentation respects the word structure of the languages it serves. It also gives you a principled critique to bring to tokenizer ablations: two tokenizers with identical compression statistics can differ substantially in how often their segments align with meaningful units, and that difference shows up downstream.

Morphological regularity in vector arithmetic. Word2Vec-style embedding spaces famously encode some regular inflectional contrasts as approximately consistent offsets, so that the offset between a regular verb's present-participle form and its past-tense form is roughly constant across verbs. Irregular pairs like go/went sit close together through contextual similarity but show no such consistent offset, because nothing in their shared surface material signals the relationship. Morphology thus sets the boundary conditions for what analogy structure in embedding spaces can and cannot capture. The offset trick is, at bottom, a discovery about regular inflection wearing the clothes of linear algebra.