Morphology and Word Formation: Morphemes, Inflection

Michael BrenndoerferMay 2, 202537 min read

Part of Language AI Handbook

Explains how words decompose into morphemes, why inflection and derivation differ, and how morphology explains NLP sparsity, OOV rates, and tokenizer failures.

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Morphology and Word Formation

Words look like atomic units when you read them on a screen, but they are anything but. The word unbelievably packs four meaningful pieces into twelve letters: un- (not), believe (the core meaning), -able (capable of being X-ed), and -ly (turns an adjective into an adverb). A speaker of English need not have memorized "unbelievably" as a unit; they can assemble it from parts they already knew. This ability to build and decompose words is called morphology, and it is one of the main reasons language is so productive: with a modest memory of roots and affixes, speakers can produce and understand millions of word forms they have never seen before.

Throughout Part V we treated words as tokens to be counted and embedded, and we saw how subword tokenization emerged from the vocabulary problem: you cannot enumerate every word form, so you break unfamiliar ones into pieces. But those pieces were chosen by frequency, not by meaning. Morphology tells us what the right pieces often are, why certain words are missing from our vocabularies and embeddings, and how a model that knows unlock can reasonably guess something about unlockable. In this chapter we build the linguistic foundation for word structure. The next chapter covers computational morphological analysis and generation in depth, and later chapters in Part VII move from the structure of words to the structure of sentences.

Morphemes: The Atoms of Word Structure

Morphology is the study of the internal structure of words. Its central unit is the morpheme.

Morpheme

A morpheme is the smallest meaning-bearing unit of a language. It may be a whole word (like dog) or a piece of one (like the plural -s or the negative prefix un-). Crucially, "smallest" refers to meaning, not size: cat has three characters but is a single morpheme, while cats has four characters but contains two.

Every word form can be analyzed as a structured combination of morphemes. Linguists distinguish a few roles these morphemes can play:

  • Root: the core morpheme carrying the word's primary lexical meaning. In unhappiness, the root is happy. Every simple (non-compound) word has exactly one root, and it is usually the piece that survives when all decoration is stripped away; compounds, covered below, combine two or more roots. Roots may themselves be free (like happy, which stands alone as a word) or bound (like the Latin-derived root -ject in project and inject, which cannot appear without an affix).
  • Affix: a bound morpheme attached to a root. Prefixes attach before (un-, re-, dis-), suffixes after (-ness, -ed, -ly), infixes inside (rare in English, common in Tagalog), and circumfixes surround the root (German ge-...-t in past participles like gemacht). Affixes are the workhorses of morphology: they carry grammatical information like tense and number, or shift meaning and category as derivation does.
  • Stem: the form a morphological operation attaches to. In unlockable, the stem for -able is the entire derived word unlock, not just the root lock. The distinction between root and stem matters because affixes do not always attach directly to roots; they often attach to already-built structures, which is why word formation is best understood as a sequence of operations rather than a flat pile of pieces.
Lexeme and Lemma

A lexeme is an abstract vocabulary item: the unit listed in a mental dictionary. Run, runs, ran, and running are all word forms of a single lexeme, RUN. A lemma is the canonical citation form chosen to represent that lexeme, conventionally the infinitive for verbs (run) and the singular for nouns (cat). Keep these distinct from the surface token, which is what appears in text. One lemma corresponds to many tokens.

Morphemes come in two flavors. Free morphemes can stand alone as words (book, green, dog). Bound morphemes cannot and must attach to something else (-s, un-, -able). English is relatively light on bound morphology, but even here the distinction shapes everything from plural handling to adjective comparison. In many other languages the balance tips heavily toward bound material, and we will see shortly how that imbalance reshapes entire NLP pipelines.

Not every decomposition a computer proposes is linguistically real. Carpet contains the letter sequence car-pet, but neither piece contributes meaning; the word is a single monomorphemic borrowing. Similarly, relish has no semantic relation to re- + lish. Morphology is about meaning-bearing units, and this constraint is what separates genuine morphological analysis from naive string matching. The test is simple to state and hard to automate: does each proposed piece carry a consistent meaning across the words it appears in? Un- means "not" or "reverse" in thousands of English words, so it is a real morpheme; car- means nothing systematic, so it is not. This gap between string similarity and semantic reality will resurface repeatedly, both in our rule-based segmenter later in this chapter and in the statistical tokenizers of Part V.

Inflection versus Derivation

The two most important morphological processes share a mechanism (attach an affix) but serve very different purposes. Keeping them distinct is essential, because most NLP tools handle them differently, and conflating them causes subtle bugs. Lemmatizers, for instance, are built to undo inflection only; pointing one at a derived word and expecting it to recover the original root misunderstands both the tool and the linguistics.

Inflection

Inflection creates different grammatical forms of the same lexeme. Inflected forms share the word's core meaning and part of speech; they differ only in grammatical features like number, tense, person, case, or gender. Think of inflection as changing the "settings" on a single lexical item rather than creating a new item altogether.

English inflection is famously sparse, but it exists:

LexemeInflected formsFeature expressed
catcat, catsnumber (singular/plural)
walkwalks, walked, walkingtense, aspect, agreement
happyhappier, happiestdegree of comparison
sheshe, her, herscase

Other languages inflect far more heavily. Spanish hablo, hablas, habla, hablamos, habláis, hablan are six forms of one verb, each marking person and number. Finnish nouns inflect for fifteen cases, and Turkish verbs chain together tense, aspect, mood, person, and number markers. We return to this variation shortly, because it has direct engineering consequences: the more inflection a language performs, the more surface forms any fixed vocabulary must either contain or fail on.

Out[4]:
Visualization
Bar chart on a log scale showing paradigm sizes growing from analytic through light, moderate, heavy, and agglutinative to polysynthetic languages.
Illustrative surface-form counts per lexeme across languages with increasing inflectional richness. Each additional grammatical category multiplies the paradigm, so a fixed vocabulary covers exponentially fewer forms as inflection grows.

Even under generous assumptions about how many categories combine, the trend in @fig-paradigm-size-typology is clear: moving one step along the typological spectrum multiplies the number of forms a system must handle by orders of magnitude. This is the arithmetic behind every vocabulary-budget failure we examine below.

The defining property of inflection is paradigmatic regularity: all forms of a lexeme slot into the same grammatical environments. Replace walked with strolled in a sentence and the grammar still holds, because both are past-tense verb forms. Inflection never changes part of speech and rarely shifts meaning beyond grammatical content. This regularity is also what makes inflection tractable to model: because every lexeme of a given class takes essentially the same set of forms, a system that learns the pattern for one verb has largely learned it for all of them, modulo irregular exceptions like go/went.

Derivation

Derivation creates new lexemes. Derived forms get their own dictionary entries because their meaning or word class departs from the base.

  • happy (adjective) →\to unhappy (adjective, negated) →\to unhappiness (noun); unhappily (adverb) branches off unhappy via -ly
  • nation (noun) →\to national (adjective) →\to nationalize (verb) →\to nationalization (noun again)

Notice the last chain: derivation can cycle a word back to its original part of speech while accumulating layers of meaning. Each derived form is a genuinely new lexical item, and some derivations are semantically idiosyncratic. Some -er derivatives are straightforwardly compositional agents (unionizer organizes workers into a union; killer kills), but the suffix is not uniformly agentive: a dinner is not an agent at all. These quirks matter for anyone building compositional models of meaning: if suffix meanings were fully predictable, deriving semantics from morphology would be trivial, and it is not.

Derivational affixes are also selective in ways inflectional affixes are not. -ness attaches to adjectives (kindness, darkness) but not to verbs; *runness is impossible. These selectional restrictions make derivation a knowledge-heavy process, which is why the next chapter treats morphological generation as a real modeling problem rather than a solved string operation. Knowing that -ness needs an adjective base is exactly the kind of structured knowledge that neither frequency counts nor character n-grams capture cleanly.

Out[6]:
Visualization
Heatmap of suffix compatibility with noun, verb, and adjective bases.
Illustrative selectivity of common English suffixes across base part-of-speech classes. Inflectional suffixes are highly productive within the relevant lexical class, while derivational suffixes vary in which classes they license.

@fig-affix-selectivity-heatmap makes the asymmetry visible. The bottom two rows, representing inflection, are bright only in the relevant base class: plural -s applies productively to count nouns, while past -ed and progressive -ing apply productively to verbs. The derivational rows show different, often narrower selectional patterns. Productivity within a class is not the same as indiscriminate attachment across parts of speech.

Telling Them Apart

In practice, ask three questions about an affixed form:

  1. Does the part of speech change? quick →\to quickly changes adjective to adverb: derivation. walk →\to walked keeps it a verb: inflection.
  2. Would the form get its own dictionary entry? Nationalize would; nationalizes would not.
  3. Is the affix productive and paradigmatic? Nearly every English verb takes -ing; almost none take -ment. That asymmetry marks -ing as inflectional and -ment as derivational.

No single question is decisive on its own, since borderline cases exist where derivational affixes approach productivity or inflectional ones drift toward lexicalization. But asked together, they resolve the vast majority of cases correctly, and they encode precisely the properties our computational tools exploit: lemmatizers target the paradigmatic, part-of-speech-preserving changes, and leave everything else alone.

Allomorph

The same morpheme can surface with different pronunciations or spellings depending on context. These variants are called allomorphs. The English plural -s is pronounced /s/ after voiceless sounds (cats), /z/ after voiced ones (dogs), and /ɪz/ after sibilants (buses). Spelling-level allomorphy includes y →\to ie (study/studies) and silent-e deletion (make/making). Allomorphy is why naive suffix stripping produces stems like studi and knive: the algorithm removes the affix correctly but cannot recover the root's original spelling.

Allomorphy deserves special attention from NLP practitioners because it sits at the exact boundary between sound, spelling, and structure. The underlying morpheme is one thing; its written realization varies with the phonetic environment around it. Any pipeline that treats spelling changes as noise will systematically corrupt stems, and any evaluation of a stemmer or segmenter should check whether allomorphic repairs are being made, not just whether affixes were removed.

Compounding

Compounding combines two or more free morphemes into a single new word: toothbrush, bookstore, laptop, database administrator. Compounding is extraordinarily productive in English, German (Donaudampfschifffahrtsgesellschaft, "Danube steamship shipping company"), and many other languages, and it generates open-ended novelty that no fixed lexicon can capture. Unlike affixation, compounding draws its raw material entirely from existing free words, which means the space of possible compounds grows at least quadratically with the vocabulary itself (faster still for multi-part compounds, and unboundedly under recursion).

Three properties matter for NLP:

  • Headedness: compounds have a head that determines the category and core meaning. A doghouse is a kind of house, not a kind of dog. The head is typically rightmost in English and leftmost in Romance languages. Semantic interpretation therefore depends on identifying the head, a task related to parsing that we revisit in the syntax chapters. Get the head wrong and you invert the meaning: treating doghouse as a kind of dog produces nonsense that no amount of embedding smoothing will repair.
  • Recursion: compounds nest. Language technology research funding is a compound built from smaller compounds, and its bracketing determines its meaning. Ambiguity is real: a Spanish teacher could be a teacher of Spanish or a teacher who is Spanish. Just as bracketing determined the meaning of unhappiness earlier, bracketing inside compounds determines what modifies what, and there is often no local cue that resolves it.
  • Orthographic instability: English writes compounds inconsistently as closed (notebook), hyphenated (mother-in-law), or open (ice cream). Whether hot dog is one word or two depends on whether lunch or dogs is under discussion. This variability directly complicates the word tokenization we studied in Part I, since a whitespace-based tokenizer silently splits open compounds and merges nothing. Worse, the same compound can migrate between styles over time or across style guides, so even a curated list of known compounds goes stale.

Other Word Formation Processes

Affixation and compounding cover most cases, but languages innovate through several additional mechanisms that show up constantly in real text. Each one stresses a different assumption that classical text-processing pipelines quietly make, so it pays to know them individually.

Cliticization attaches a bound element that behaves syntactically like a word. English possessive 's is the classic case: in the queen of England's crown, 's attaches to England even though it marks possession by the queen. Contractions like we're and they've blend clitics with the tokenization questions from Part I: is we're one token, two, or a word plus a clitic? Clitics occupy an awkward middle ground between morphology and syntax, and different toolkits answer the tokenization question differently, which silently changes your corpus statistics.

Conversion (zero derivation) changes a word's part of speech without changing its form. Google became a verb; a run came from the verb run. Conversion is invisible to string-level processing, which is why part-of-speech taggers from Part VI must rely on context rather than word shape. It is also increasingly common in modern text as brands, products, and technologies get verbed (to email, to zoom, to dm), so any lexicon-based tagger ages quickly.

Blending fuses fragments of words: smog (smoke + fog), brunch, motel. Acronymy builds words from initials: NATO, laser, FOMO. Both are common sources of out-of-vocabulary tokens in modern text, and both are essentially unpredictable in advance: nobody could have enumerated FOMO before the culture that produced it existed.

Back-formation removes a perceived affix: edit was extracted from editor, burgle from burglar. Back-formation is historically fascinating because it runs morphology backwards: speakers assume a longer word is derived and invent the shorter "original." Reduplication repeats all or part of a word for emphasis or plurality, and is grammatically central in Indonesian, Tagalog, and many other languages. For those languages reduplication is not a curiosity but a load-bearing grammatical mechanism, and pipelines that ignore it lose real information.

Finally, root-and-pattern (templatic) morphology interleaves vowels into consonantal roots. Semitic languages like Arabic and Hebrew store roots as bare consonant skeletons: Arabic k-t-b carries the notion of writing, and vowel patterns derive kataba (he wrote), kitāb (book), kuttāb (writers), maktab (office). The root's consonants are interleaved with vowels and almost never appear as one contiguous substring, which breaks character n-gram heuristics and makes templatic languages a standing challenge for subword tokenizers, as we discussed in the Tokenization Challenges chapter of Part V. Where European-style morphology lets a tokenizer find the root by trimming edges, templatic morphology hides it in the middle of the word, interleaved with unrelated vowels, and no amount of edge-trimming recovers it.

Morphological Typology: Why Language Choice Changes Your Pipeline

Languages differ systematically in how much work their morphology does, and this typology predicts, surprisingly well, which NLP techniques will struggle. The categories below form a spectrum rather than a set of boxes, and most languages sit somewhere in the middle, but the extremes clarify the trade-offs sharply.

Analytic (isolating) languages, like Mandarin Chinese and Vietnamese, pack little into each word. Grammatical information lives in word order and separate particles, so a large fraction of word forms are monomorphemic. Vocabulary is manageable, but syntax carries the load. Text-processing effort shifts accordingly: segmentation of continuous script becomes the hard problem, while inflection handling nearly vanishes.

Agglutinative languages, like Turkish, Finnish, Hungarian, and Swahili, string together long sequences of clearly delimited affixes, each contributing one meaning. Turkish evlerimizden decomposes cleanly as ev-ler-imiz-den: house, plural, our, from ("from our houses"). The combinatorics explode: Turkish verb paradigms contain millions of theoretically possible forms, so enumerating them in a vocabulary is hopeless, but rule-based decomposition works well because the affix boundaries are regular. Agglutinative languages are thus simultaneously the worst case for vocabularies and the best case for explicit morphological modeling.

Fusional languages, like Russian and Latin, compress multiple grammatical meanings into a single unsegmentable ending (Arabic layers this kind of fusion on top of the templatic root-and-pattern system described above). Russian книга, книги, книге, книгу encode case and number jointly in endings you cannot cleanly split into one-morpheme-one-meaning pieces. Paradigm tables, not string rules, traditionally describe these systems. Fusion defeats the natural decomposition strategy, so systems for fusional languages tend to lean on lookup tables and learned classifiers over whole endings.

Polysynthetic languages, like Inuktitut and Mohawk, push fusion to extremes, packing what English expresses with a full clause into a single word. At this end of the spectrum the very notion of a "word" as a processing unit comes into question, since one word may warrant an entire parse tree of its own.

The engineering lesson is direct. Recall the vocabulary problem and its subword solutions from Part V. For English, a 30,000-item vocabulary covers most running text comfortably. Apply the same budget to Turkish or Finnish and a large share of tokens fall back to tiny fragments, inflating sequence lengths and diluting each fragment's representation. Longer sequences mean more positions for attention to cover, more memory per document, and weaker signal per token. This is one reason multilingual models allocate vocabulary unequally across languages, and why morphologically aware tokenization remains an active research area.

Why Word Structure Matters for Language Models

It is worth pausing to trace exactly where morphology enters the pipelines you have already built. None of the following points requires new machinery; each is a familiar failure mode whose root cause turns out to be linguistic.

Statistical sparsity. N-gram language models estimate probabilities by counting. Inflection multiplies events: instead of learning one distribution over walk, you learn separate distributions over walk, walks, walked, and walking, and your counts for each thin dramatically. Smoothing techniques from Part II mitigate this, but they are patches over a structural problem: the model has no way to know that walked and strolled behave similarly because both are past-tense forms. Smoothing redistributes probability mass according to generic assumptions; morphology would redistribute it according to the actual generative structure of the data.

Out-of-vocabulary rates. Any word-form-level vocabulary misses rare inflections, novel derivations, and fresh compounds. Morphology explains both why these forms exist (productivity) and what they mean given their parts, which is precisely the knowledge a subword tokenizer approximates statistically. When you encounter an unseen form like unputdownable, morphology tells you it is comprehensible in principle; a closed vocabulary simply reports a miss.

Embedding coverage. As we saw in the FastText chapter of Part IV, representing words as bags of character n-grams lets rare words borrow strength from their substrings: superstitious shares n-grams with superstition. FastText works precisely because character n-grams approximate morphemes. When the approximation aligns with true morphological structure, embeddings generalize well; when it does not, they inherit noise from unrelated words. The n-gram bag has no way to prefer super- over coincidental overlaps, so its generalizations are only as good as the accidental alignment between substrings and meaning.

Subwords as implicit morphology. BPE, WordPiece, and Unigram tokenization learn frequent substrings, and in practice many learned tokens correspond to real morphemes: common prefixes, suffixes, and stems survive as units. But the correspondence is loose. Frequency-driven merging often preserves frequent strings across word boundaries (as with there inside therefore), fragments rare affixes inconsistently across similar words, and carves words like Wednesday into chunks bearing no semantic relation to their parts. Understanding morphology lets you diagnose these failures and evaluate whether a tokenizer's segmentation respects the word structure of the languages it serves. It also gives you a principled critique to bring to tokenizer ablations: two tokenizers with identical compression statistics can differ substantially in how often their segments align with meaningful units, and that difference shows up downstream.

Morphological regularity in vector arithmetic. Word2Vec-style embedding spaces famously encode some regular inflectional contrasts as approximately consistent offsets, so that the offset between a regular verb's present-participle form and its past-tense form is roughly constant across verbs. Irregular pairs like go/went sit close together through contextual similarity but show no such consistent offset, because nothing in their shared surface material signals the relationship. Morphology thus sets the boundary conditions for what analogy structure in embedding spaces can and cannot capture. The offset trick is, at bottom, a discovery about regular inflection wearing the clothes of linear algebra.

Worked Example: Parsing Unhappiness

Let us analyze one word carefully, because a single example exposes nearly every issue in this chapter: affix identification, allomorphic repair, hierarchical structure, the inflection/derivation distinction, and ambiguity.

Consider unhappiness. A native speaker parses it effortlessly, but let us be explicit about the reasoning.

Step 1: Identify bound morphemes. The initial un- and final -ness are recognizable affixes. Strip them tentatively and we are left with happi. Tentativeness matters here: stripping is a hypothesis, not a conclusion, and the next step exists precisely to check it.

Step 2: Repair allomorphy. Happi is not an English word. The spelling change y →\to i occurred when -ness attached. Undoing it yields happy, a real free morpheme and the root. This step is where purely mechanical stemmers fail, as we will confirm experimentally below.

Step 3: Order the operations. The structure is nested, not flat: un- applied to happy produced unhappy, and then -ness applied to unhappy produced unhappiness. The correct bracketing is [[un-happy]-ness]. Flattening this to [un-[happy-ness]] would misrepresent the semantics: the negation scopes over the adjective, not the noun. There is no noun *unhapp. Word formation, like sentence formation, has a tree structure, and getting the tree wrong gets the meaning wrong.

Step 4: Classify each operation. un- + adjective →\to adjective is derivation (it creates a new lexeme). -ness + adjective →\to noun is derivation. Neither operation is inflection, so unhappiness is a distinct lexeme from happy, and a lemmatizer should leave it untouched.

Now contrast with unhappier. Same prefix, similar suffix, completely different analysis. Here -er is inflectional (comparative degree), so unhappier is merely a form of the lexeme UNHAPPY, and its lemma is unhappy. Surface similarity hides a categorical distinction that downstream tools must respect: lemmatizing unhappiness should change nothing, while lemmatizing unhappier must return unhappy. Two words differing by a couple of letters demand opposite treatments, which is exactly why lexicon-free string rules struggle here.

One more twist: ambiguity. Consider undoable. Is it [un-do]-able, "able to be undone," or un-[do-able], "not able to be done"? Both parses are attested, and they are near antonyms. Prefix un- attaches to verbs (meaning "reverse") and to adjectives (meaning "not"), so the parse depends on knowing whether the stem is a verb or an adjective. Morphological ambiguity is systematic, not pathological, and resolving it requires the same disambiguation machinery as other linguistic ambiguities: context. A sentence like "the report was undoable" and one like "the knot was undoable" pull toward opposite readings, and no property of the string alone settles the matter.

Code Implementation

We will now build up these intuitions computationally. First, a comparison of stemming and lemmatization on words chosen to stress irregularity and derivation, then a small rule-based segmenter that reproduces our worked example, and finally a look at how a frequency-driven BPE tokenizer segments the same words compared to a morphological analysis. Each experiment is designed to make one theoretical point concrete: allomorphy breaks string rules, lexicons fix inflection but not derivation, and frequency-driven fragmentation diverges from meaning exactly where morphology is most productive.

Setup

We use NLTK for corpora, the Porter stemmer, and the WordNet lemmatizer, and Hugging Face's tokenizers library for BPE. If you do not have tokenizers, install it with uv pip install tokenizers.

In[7]:
Code
import nltk

for pkg in ["punkt", "wordnet", "omw-1.4", "gutenberg"]:
    nltk.download(pkg, quiet=True)

from collections import Counter

from nltk.corpus import gutenberg
from nltk.stem import PorterStemmer, WordNetLemmatizer

Stemming versus Lemmatization

Recall the distinction: a stemmer crudely chops affixes with string rules, while a lemmatizer looks the word up and returns a real dictionary lemma, using part-of-speech information to disambiguate. We probe both with nine words covering regular inflection, irregular inflection, and derivation.

In[9]:
Code
stemmer = PorterStemmer()
lemmatizer = WordNetLemmatizer()

## (surface form, WordNet POS for lemmatization)
test_words = [
    ("running", "v"),
    ("ran", "v"),  # regular vs irregular verb
    ("studies", "n"),
    ("knives", "n"),  # y->i and f->ves allomorphy
    ("geese", "n"),
    ("mice", "n"),  # suppletive plurals
    ("better", "a"),
    ("happier", "a"),  # comparative: irregular vs regular
    ("unhappiness", "n"),  # derived noun (its own lemma)
]

stem_results = [(w, stemmer.stem(w)) for w, _ in test_words]
lemma_results = [(w, lemmatizer.lemmatize(w, pos=p)) for w, p in test_words]
Out[10]:
Console
word           Porter stem    WN lemma      
------------------------------------------
running        run            run           
ran            ran            run           
studies        studi          study         
knives         knive          knife         
geese          gees           goose         
mice           mice           mouse         
better         better         good          
happier        happier        happy         
unhappiness    unhappi        unhappiness

Look closely at the failures, because each one is a lesson. The stemmer mangles studies into studi and knives into knive: it strips the affix correctly but cannot reverse allomorphic spelling changes. It leaves ran, mice, better, and happier untouched, since their affixes either do not exist or fail the stemmer's measure condition; it even shaves the silent e off geese, yielding gees. The lemmatizer handles all of these perfectly because WordNet stores irregular paradigms explicitly. On unhappiness, however, the lemmatizer (correctly, linguistically) returns the word unchanged: derivation is not inflection, and lemmatizers target inflection only. Meanwhile the stemmer over-applies, chopping it to unhappi and erasing the derivational structure. Note that both tools are doing something defensible by their own lights; the failures come from asking a tool designed for one job to do another.

Let us score both methods against the true lemma of each word and visualize the gap.

In[11]:
Code
expected_lemma = {
    "running": "run",
    "ran": "run",
    "studies": "study",
    "knives": "knife",
    "geese": "goose",
    "mice": "mouse",
    "better": "good",
    "happier": "happy",
    "unhappiness": "unhappiness",
}

stem_correct = sum(out == expected_lemma[w] for w, out in stem_results)
lemma_correct = sum(out == expected_lemma[w] for w, out in lemma_results)

scores = {"Porter stemmer": stem_correct, "WordNet lemmatizer": lemma_correct}
total = len(test_words)
Out[12]:
Visualization
Bar chart comparing correct lemma recovery for Porter stemmer versus WordNet lemmatizer.
Number of test words reduced to the correct lemma by the Porter stemmer versus the WordNet lemmatizer (9 words total). The stemmer recovers 1 of 9, losing points on irregular forms, allomorphic spelling, and derived words; the lemmatizer recovers all 9.

The lemmatizer wins decisively here, but remember the trade-off: lemmatization depends on a lexicon and benefits from part-of-speech information, so it is slower and impractical for languages lacking machine-readable dictionaries, while stemming is fast and lexicon-free, though its rules must be rewritten for each language. Neither approach touches derivation, which is why morphological analysis proper deserves its own machinery (next chapter).

It is also instructive to see where the points were lost, not just how many.

In[13]:
Code
from book_plot_style import PALETTE

words_order = [w for w, _ in test_words]
correct_flags = [stemmer.stem(w) == expected_lemma[w] for w in words_order]

failure_type = {
    "running": "regular inflection",
    "ran": "irregular form",
    "studies": "allomorphy",
    "knives": "allomorphy",
    "geese": "irregular form",
    "mice": "irregular form",
    "better": "irregular form",
    "happier": "regular inflection",
    "unhappiness": "derivation",
}
type_colors = {
    "regular inflection": PALETTE["primary"],
    "irregular form": PALETTE["coral"],
    "allomorphy": PALETTE["amber"],
    "derivation": PALETTE["purple"],
}
Out[14]:
Visualization
Grouped bar chart showing which test words the Porter stemmer got right and wrong, colored by failure type.
Per-word correctness of the Porter stemmer against the expected lemma, grouped by failure type: irregular forms sharing no strippable affix, allomorphic spelling changes, and derived words a stemmer should arguably leave alone. Losses concentrate in the irregular and allomorphic categories, exactly where lexicon-free string rules cannot reach.

@fig-stemming-error-breakdown shows that the stemmer's losses concentrate in the irregular and allomorphic columns, exactly the categories a lexicon-free rule cannot reach, while the single derivation failure reflects a category error rather than a string error.

A Rule-Based Segmenter

Our worked example followed a procedure: strip affixes, repair spelling, check for real morphemes. Let us implement a miniature version with explicit prefix and suffix inventories and see it reproduce the unhappiness analysis, along with its characteristic failure modes.

In[15]:
Code
PREFIXES = ["anti", "dis", "mis", "over", "under", "re", "un"]
SUFFIXES = [
    "ational",
    "ization",
    "iveness",
    "fulness",
    "ability",
    "ments",
    "ness",
    "ment",
    "tion",
    "ings",
    "ing",
    "ers",
    "er",
    "est",
    "ed",
    "ly",
    "es",
    "s",
]


def repair_stem(stem):
    """Reverse common suffixation spelling changes."""
    if stem.endswith("i"):
        candidate = stem[:-1] + "y"  # happi -> happy, studi -> study
        return candidate
    if stem.endswith("at") and len(stem) > 3:
        return stem  # creat(e)ion patterns left as-is
    return stem


def segment_word(word):
    """Greedy prefix/suffix stripping with spelling repair."""
    parts, remainder = [], word
    for pref in PREFIXES:
        if remainder.startswith(pref) and len(remainder) > len(pref) + 2:
            parts.append(pref)
            remainder = remainder[len(pref) :]
            break
    for suf in SUFFIXES:
        if remainder.endswith(suf) and len(remainder) > len(suf) + 2:
            stem = remainder[: len(remainder) - len(suf)]
            parts.extend([repair_stem(stem), suf])
            return parts
    parts.append(remainder)
    return parts


test_cases = [
    "unhappiness",
    "unhappier",
    "studying",
    "nationalization",
    "carpet",
]
segments = {w: segment_word(w) for w in test_cases}
Out[16]:
Console
unhappiness        -> un + happy + ness
unhappier          -> un + happy + er
studying           -> study + ing
nationalization    -> national + ization
carpet             -> carpet

The segmenter gets the headline cases right: unhappiness decomposes into un + happy + ness after repairing the y-to-i spelling change, matching our manual analysis. But observe the close call with carpet: it survives unsplit only because pet is absent from the suffix inventory, and a slightly richer inventory would split it into car + pet despite the word being monomorphemic. String rules have no access to meaning, so they cannot distinguish genuine morphemes from coincidental substrings. Real systems address this with a lexicon of licensed roots and affixes, or by learning segmentations from distributional evidence, as Morfessor-style unsupervised methods do. We will examine those approaches in the next chapter.

What a Frequency-Driven Tokenizer Does Instead

Now let us see what BPE, the algorithm behind GPT-style tokenizers from Part V, does with these same words. We train a deliberately tiny BPE model (a 250-token vocabulary) on a sample of Melville's Moby Dick. A small vocabulary forces heavy fragmentation, which mimics what happens to morphologically complex languages under production-sized budgets.

In[17]:
Code
from tokenizers import Tokenizer, models, pre_tokenizers, trainers

raw_text = " ".join(gutenberg.raw("melville-moby_dick.txt").split())
with open("moby_train_sample.txt", "w") as f:
    f.write(raw_text[:400_000])

bpe = Tokenizer(models.BPE())
bpe.pre_tokenizer = pre_tokenizers.Whitespace()
trainer = trainers.BpeTrainer(vocab_size=250, initial_alphabet=[])
bpe.train(["moby_train_sample.txt"], trainer=trainer)
Out[18]:
Console
whales         -> ['whal', 'es']
swimming       -> ['s', 'w', 'im', 'm', 'ing']
unhappiness    -> ['un', 'h', 'ap', 'p', 'in', 'ess']
boats          -> ['bo', 'at', 's']
harpooner      -> ['h', 'ar', 'po', 'on', 'er']

With its vocabulary capped at 250 tokens, the tokenizer has learned whatever substrings happened to be frequent in whaling-era prose, and it fragments everything else into character-level leftovers. The trainer's vocab_size is a cap on the total learned vocabulary, including the initial alphabet and any special tokens; it is not the number of merge operations. Notice that the fragments respect frequency, not meaning: a morpheme like un- survives or shatters depending on how often words beginning with un- appeared in the training text, not on whether it carries a consistent meaning. Scale the vocabulary up to modern sizes and the fit improves, but the principle holds. Subword tokenization approximates morphology statistically, and the approximation degrades precisely where morphology is most productive: rare, novel, and complex words.

Let me quantify that degradation. We compare three groups of words: common bases, mostly regularly inflected or derived forms of those bases, and novel-looking coinages assembled from familiar morphemes that no tokenizer has ever seen.

In[19]:
Code
base_words = [
    "whale",
    "boat",
    "sea",
    "ship",
    "water",
    "captain",
    "harpoon",
    "ocean",
    "crew",
    "wind",
]
formed_words = [
    "whales",
    "boating",
    "seaward",
    "shipping",
    "waterlogged",
    "captains",
    "harpoons",
    "oceanic",
    "crewing",
    "winds",
]
novel_words = [
    "whaleness",
    "boatify",
    "seawardsman",
    "shiphood",
    "wateristic",
    "captainless",
    "harpoonerly",
    "oceanite",
    "crewdom",
    "windageous",
]

groups = {
    "Common bases": base_words,
    "Formed words": formed_words,
    "Novel words": novel_words,
}
avg_subword_counts = {
    name: sum(len(bpe.encode(w).tokens) for w in words) / len(words)
    for name, words in groups.items()
}
Out[20]:
Visualization
Bar chart of average subword token counts rising from base to formed to novel words.
Average number of BPE subword tokens per word for common bases, regularly formed words, and morphologically well-formed novel words, under a 250-token BPE vocabulary. Fragmentation rises steadily from frequent bases to novel words, showing that merge decisions track corpus frequency rather than morphological structure.

The trend is exactly what morphological theory predicts. Common bases fragment least because they are frequent enough for BPE to have merged them early. Formed words fragment more, and novel words, though built entirely from familiar morphemes, fragment worst of all, since their surface forms never contributed to any merge decision. A human reader decodes captainless instantly from known parts. The tokenizer sees an alien string. This gap between compositional meaning and statistical fragmentation is the core argument for bringing morphological knowledge into tokenization.

The gap should narrow as the vocabulary grows, but it never closes completely. Let us verify this directly by retraining BPE at a range of vocabulary sizes and tracking fragmentation for each group.

In[21]:
Code
vocab_sizes = [150, 250, 500, 1000, 2000, 3000]
frag_by_vocab = {name: [] for name in groups}
for v in vocab_sizes:
    tok_v = Tokenizer(models.BPE())
    tok_v.pre_tokenizer = pre_tokenizers.Whitespace()
    tok_v.train(
        ["moby_train_sample.txt"],
        trainers.BpeTrainer(vocab_size=v, initial_alphabet=[]),
    )
    for name, words in groups.items():
        frag_by_vocab[name].append(
            sum(len(tok_v.encode(w).tokens) for w in words) / len(words)
        )
Out[22]:
Visualization
Line plot of average subword tokens per word versus BPE vocabulary size for three word groups.
Average BPE subword tokens per word for the three word groups as the vocabulary-size cap grows from 150 to 3000 tokens. Fragmentation falls everywhere, but novel words remain systematically more fragmented even at the largest budget.

@fig-bpe-fragmentation-vs-vocab-size confirms the theoretical prediction: larger vocabulary budgets buy better coverage for everyone, but the ordering of the curves never changes. Statistical tokenization compresses what is frequent; morphology is precisely the machinery for handling what is not.

One Lemma, Many Tokens

To close the loop, let us measure the phenomenon that motivates everything above: how a single lexeme scatters across many surface forms in real text. We count the forms of the verb SAY in Moby Dick, which happens to include both a regular -s/-ing paradigm and the suppletive past tense said.

In[23]:
Code
moby_tokens = [w.lower() for w in gutenberg.words("melville-moby_dick.txt")]
moby_freq = Counter(moby_tokens)

say_family = ["say", "says", "said", "saying"]
family_counts = {form: moby_freq[form] for form in say_family}
family_total = sum(family_counts.values())
distinct_forms = sum(1 for c in family_counts.values() if c > 0)
Out[24]:
Console
said       304 occurrences (47.9% of family)
say        244 occurrences (38.5% of family)
says        46 occurrences (7.3% of family)
saying      40 occurrences (6.3% of family)

4 distinct surface forms share the lexeme SAY (634 total occurrences)
Out[25]:
Visualization
Bar chart showing said dominating the counts of say, says, and saying.
Corpus frequencies of the four surface forms of the lexeme SAY in Moby Dick. The suppletive past tense 'said' dominates, illustrating how inflection scatters a lexeme's evidence across unevenly distributed forms.

Here is the practical consequence. An N-gram language model, as built in Part II, learns said and says as unrelated symbols with wildly different statistics, even though they differ only in tense. Every count for said teaches it nothing about says. Multiply this scattering across a whole vocabulary, and inflection becomes a leading source of sparsity, the very problem our smoothing techniques fought to contain. A model with access to morphology, whether through lemmatization before counting or through morphologically informed representations, pools the evidence of all four forms under one lexeme and generalizes from regular patterns to unseen combinations.

Limitations and Impact

Morphology gave classical NLP its most effective lever against vocabulary explosion, and it continues to shape modern systems. Stemming and lemmatization improved IR recall, sometimes substantially depending on the language and collection, since searching for whales should match documents containing whaling; finite-state morphological analyzers, built on the regular expression foundations of Part I, still power spell-checking, hyphenation, and machine translation for morphologically rich languages; and subword tokenization, whatever its statistical character, is a direct descendant of the insight that words decompose into reusable parts.

Yet the limitations are real, and honest treatment of them clarifies what modern models absorb implicitly and what they still lack. Rule-based segmentation fails exactly where meaning begins: our segmenter spares carpet only because pet is absent from its suffix inventory, and a slightly richer inventory would split it despite the word being monomorphemic; no string rule can know that transmission in an automotive corpus has nothing to do with trans- + mit. Supervised analyzers require annotated lexicons; broad resources such as UniMorph now reach on the order of a hundred languages, but depth and reliability thin out quickly outside a few dozen well-studied ones, leaving much of the world's linguistic diversity to unsupervised or neural approximations whose quality is hard to verify. Lemmatization helps bag-of-words models but destroys distinctions those models sometimes need: whenever it recognizes an archaic form, it collapses the deliberate archaism of thou speakest into ordinary you speak, erasing an agreement contrast that stylistic analysis may depend on. And for large pretrained models, much of everyday inflectional morphology is absorbed into scale: a sufficiently large transformer often sees enough examples of rare forms to infer their behavior from context. But scale is expensive and unequal. Low-resource languages, templatic morphology, and genuinely novel coinages remain places where the statistical shortcut thins out, which is why morphological robustness keeps resurfacing as an evaluation concern and why understanding word structure remains part of the working toolkit even in the era of trillion-token pre-training.

There is also a conceptual payoff that transcends any particular technique. Morphology teaches that linguistic form is compositional and layered: meanings combine according to systematic, partially exception-ridden rules. Holding a parsed structure in mind, with its bracketings, features, and ambiguities, prepares you for the move from words to phrases in the coming chapters. Syntax asks the same questions morphology does, one level up.

Summary

  • A morpheme is the smallest meaning-bearing unit; words are structured combinations of roots and affixes, and a word's lemma is the citation form of its lexeme.
  • Inflection creates grammatical variants of one lexeme (walk/walked/walking); derivation creates new lexemes (happy/unhappiness). Lemmatizers handle inflection only.
  • Compounding joins free morphemes productively, with headedness determining meaning and inconsistent orthography complicating tokenization.
  • Languages range from analytic through agglutinative and fusional to polysynthetic; typology predicts how badly fixed vocabularies fail.
  • Morphology explains key NLP pain points: n-gram sparsity, out-of-vocabulary rates, embedding gaps, and subword tokenizers that fragment by frequency rather than meaning.
  • Stems are chopped by rules and fail on allomorphy and irregularity; lemmas depend on lexicons and benefit from part-of-speech information; neither correctly resolves derivation or morphological ambiguity like undoable.

In the next chapter we turn this foundation into algorithms: finite-state morphology, supervised and unsupervised morphological segmentation, and the analysis and generation of word forms at scale.

Quiz

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about morphology and word formation.

Morphology and Word Formation

Question 1 of 80 of 8 completed
What is a morpheme?

Comments

No comments yet. Be the first to share your thoughts!

Reference

Citation details

Cite or share this article.

BIBTEXAcademic
@misc{brenndoerfer2025morphologyword, author = {Michael Brenndoerfer}, title = {Morphology and Word Formation: Morphemes, Inflection}, year = {2025}, url = {https://mbrenndoerfer.com/writing/morphology-word-formation-language-models}, organization = {mbrenndoerfer.com}, note = {Accessed: 2026-10-13} }
APAAcademic
Michael Brenndoerfer (2025). Morphology and Word Formation: Morphemes, Inflection. Retrieved from https://mbrenndoerfer.com/writing/morphology-word-formation-language-models
MLAAcademic
Michael Brenndoerfer. "Morphology and Word Formation: Morphemes, Inflection." 2026. Web. October 13, 2026. <https://mbrenndoerfer.com/writing/morphology-word-formation-language-models>.
CHICAGOAcademic
Michael Brenndoerfer. "Morphology and Word Formation: Morphemes, Inflection." Accessed October 13, 2026. https://mbrenndoerfer.com/writing/morphology-word-formation-language-models.
HARVARDAcademic
Michael Brenndoerfer (2025) 'Morphology and Word Formation: Morphemes, Inflection'. Available at: https://mbrenndoerfer.com/writing/morphology-word-formation-language-models (Accessed: October 13, 2026).
SimpleBasic
Michael Brenndoerfer (2025). Morphology and Word Formation: Morphemes, Inflection. https://mbrenndoerfer.com/writing/morphology-word-formation-language-models

About the author

Continue with the full handbook

This chapter is part of Language AI Handbook. Use the handbook page to browse the complete table of contents and continue reading in sequence.

Explore Language AI Handbook
Newsletter

Stay up to date

Get articles, book updates, and news delivered to your inbox.

No spam, unsubscribe anytime.

or

Join the community

Sign in to remove popups, track your reading progress, and join the discussion.