Morphology and Word Formation › Limitations and Impact

Morphology gave classical NLP its most effective lever against vocabulary explosion, and it continues to shape modern systems. Stemming and lemmatization improved IR recall, sometimes substantially depending on the language and collection, since searching for whales should match documents containing whaling; finite-state morphological analyzers, built on the regular expression foundations of Part I, still power spell-checking, hyphenation, and machine translation for morphologically rich languages; and subword tokenization, whatever its statistical character, is a direct descendant of the insight that words decompose into reusable parts.

Yet the limitations are real, and honest treatment of them clarifies what modern models absorb implicitly and what they still lack. Rule-based segmentation fails exactly where meaning begins: our segmenter spares carpet only because pet is absent from its suffix inventory, and a slightly richer inventory would split it despite the word being monomorphemic; no string rule can know that transmission in an automotive corpus has nothing to do with trans- + mit. Supervised analyzers require annotated lexicons; broad resources such as UniMorph now reach on the order of a hundred languages, but depth and reliability thin out quickly outside a few dozen well-studied ones, leaving much of the world's linguistic diversity to unsupervised or neural approximations whose quality is hard to verify. Lemmatization helps bag-of-words models but destroys distinctions those models sometimes need: whenever it recognizes an archaic form, it collapses the deliberate archaism of thou speakest into ordinary you speak, erasing an agreement contrast that stylistic analysis may depend on. And for large pretrained models, much of everyday inflectional morphology is absorbed into scale: a sufficiently large transformer often sees enough examples of rare forms to infer their behavior from context. But scale is expensive and unequal. Low-resource languages, templatic morphology, and genuinely novel coinages remain places where the statistical shortcut thins out, which is why morphological robustness keeps resurfacing as an evaluation concern and why understanding word structure remains part of the working toolkit even in the era of trillion-token pre-training.

There is also a conceptual payoff that transcends any particular technique. Morphology teaches that linguistic form is compositional and layered: meanings combine according to systematic, partially exception-ridden rules. Holding a parsed structure in mind, with its bracketings, features, and ambiguities, prepares you for the move from words to phrases in the coming chapters. Syntax asks the same questions morphology does, one level up.