Morphology and Word Formation › Code Implementation
Language AI HandbookAll chapters Morphology and Word Formation › Code ImplementationOne section · Sections in this chapter9 / 12 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
We will now build up these intuitions computationally. First, a comparison of stemming and lemmatization on words chosen to stress irregularity and derivation, then a small rule-based segmenter that reproduces our worked example, and finally a look at how a frequency-driven BPE tokenizer segments the same words compared to a morphological analysis. Each experiment is designed to make one theoretical point concrete: allomorphy breaks string rules, lexicons fix inflection but not derivation, and frequency-driven fragmentation diverges from meaning exactly where morphology is most productive.
Setup
We use NLTK for corpora, the Porter stemmer, and the WordNet lemmatizer, and Hugging Face's tokenizers library for BPE. If you do not have tokenizers, install it with uv pip install tokenizers.
import nltk
for pkg in ["punkt", "wordnet", "omw-1.4", "gutenberg"]:
nltk.download(pkg, quiet=True)
from collections import Counter
from nltk.corpus import gutenberg
from nltk.stem import PorterStemmer, WordNetLemmatizerStemming versus Lemmatization
Recall the distinction: a stemmer crudely chops affixes with string rules, while a lemmatizer looks the word up and returns a real dictionary lemma, using part-of-speech information to disambiguate. We probe both with nine words covering regular inflection, irregular inflection, and derivation.
stemmer = PorterStemmer()
lemmatizer = WordNetLemmatizer()
## (surface form, WordNet POS for lemmatization)
test_words = [
("running", "v"),
("ran", "v"), # regular vs irregular verb
("studies", "n"),
("knives", "n"), # y->i and f->ves allomorphy
("geese", "n"),
("mice", "n"), # suppletive plurals
("better", "a"),
("happier", "a"), # comparative: irregular vs regular
("unhappiness", "n"), # derived noun (its own lemma)
]
stem_results = [(w, stemmer.stem(w)) for w, _ in test_words]
lemma_results = [(w, lemmatizer.lemmatize(w, pos=p)) for w, p in test_words]word Porter stem WN lemma ------------------------------------------ running run run ran ran run studies studi study knives knive knife geese gees goose mice mice mouse better better good happier happier happy unhappiness unhappi unhappiness
Look closely at the failures, because each one is a lesson. The stemmer mangles studies into studi and knives into knive: it strips the affix correctly but cannot reverse allomorphic spelling changes. It leaves ran, mice, better, and happier untouched, since their affixes either do not exist or fail the stemmer's measure condition; it even shaves the silent e off geese, yielding gees. The lemmatizer handles all of these perfectly because WordNet stores irregular paradigms explicitly. On unhappiness, however, the lemmatizer (correctly, linguistically) returns the word unchanged: derivation is not inflection, and lemmatizers target inflection only. Meanwhile the stemmer over-applies, chopping it to unhappi and erasing the derivational structure. Note that both tools are doing something defensible by their own lights; the failures come from asking a tool designed for one job to do another.
Let us score both methods against the true lemma of each word and visualize the gap.
expected_lemma = {
"running": "run",
"ran": "run",
"studies": "study",
"knives": "knife",
"geese": "goose",
"mice": "mouse",
"better": "good",
"happier": "happy",
"unhappiness": "unhappiness",
}
stem_correct = sum(out == expected_lemma[w] for w, out in stem_results)
lemma_correct = sum(out == expected_lemma[w] for w, out in lemma_results)
scores = {"Porter stemmer": stem_correct, "WordNet lemmatizer": lemma_correct}
total = len(test_words)
The lemmatizer wins decisively here, but remember the trade-off: lemmatization depends on a lexicon and benefits from part-of-speech information, so it is slower and impractical for languages lacking machine-readable dictionaries, while stemming is fast and lexicon-free, though its rules must be rewritten for each language. Neither approach touches derivation, which is why morphological analysis proper deserves its own machinery (next chapter).
It is also instructive to see where the points were lost, not just how many.
from book_plot_style import PALETTE
words_order = [w for w, _ in test_words]
correct_flags = [stemmer.stem(w) == expected_lemma[w] for w in words_order]
failure_type = {
"running": "regular inflection",
"ran": "irregular form",
"studies": "allomorphy",
"knives": "allomorphy",
"geese": "irregular form",
"mice": "irregular form",
"better": "irregular form",
"happier": "regular inflection",
"unhappiness": "derivation",
}
type_colors = {
"regular inflection": PALETTE["primary"],
"irregular form": PALETTE["coral"],
"allomorphy": PALETTE["amber"],
"derivation": PALETTE["purple"],
}
@fig-stemming-error-breakdown shows that the stemmer's losses concentrate in the irregular and allomorphic columns, exactly the categories a lexicon-free rule cannot reach, while the single derivation failure reflects a category error rather than a string error.
A Rule-Based Segmenter
Our worked example followed a procedure: strip affixes, repair spelling, check for real morphemes. Let us implement a miniature version with explicit prefix and suffix inventories and see it reproduce the unhappiness analysis, along with its characteristic failure modes.
PREFIXES = ["anti", "dis", "mis", "over", "under", "re", "un"]
SUFFIXES = [
"ational",
"ization",
"iveness",
"fulness",
"ability",
"ments",
"ness",
"ment",
"tion",
"ings",
"ing",
"ers",
"er",
"est",
"ed",
"ly",
"es",
"s",
]
def repair_stem(stem):
"""Reverse common suffixation spelling changes."""
if stem.endswith("i"):
candidate = stem[:-1] + "y" # happi -> happy, studi -> study
return candidate
if stem.endswith("at") and len(stem) > 3:
return stem # creat(e)ion patterns left as-is
return stem
def segment_word(word):
"""Greedy prefix/suffix stripping with spelling repair."""
parts, remainder = [], word
for pref in PREFIXES:
if remainder.startswith(pref) and len(remainder) > len(pref) + 2:
parts.append(pref)
remainder = remainder[len(pref) :]
break
for suf in SUFFIXES:
if remainder.endswith(suf) and len(remainder) > len(suf) + 2:
stem = remainder[: len(remainder) - len(suf)]
parts.extend([repair_stem(stem), suf])
return parts
parts.append(remainder)
return parts
test_cases = [
"unhappiness",
"unhappier",
"studying",
"nationalization",
"carpet",
]
segments = {w: segment_word(w) for w in test_cases}unhappiness -> un + happy + ness unhappier -> un + happy + er studying -> study + ing nationalization -> national + ization carpet -> carpet
The segmenter gets the headline cases right: unhappiness decomposes into un + happy + ness after repairing the y-to-i spelling change, matching our manual analysis. But observe the close call with carpet: it survives unsplit only because pet is absent from the suffix inventory, and a slightly richer inventory would split it into car + pet despite the word being monomorphemic. String rules have no access to meaning, so they cannot distinguish genuine morphemes from coincidental substrings. Real systems address this with a lexicon of licensed roots and affixes, or by learning segmentations from distributional evidence, as Morfessor-style unsupervised methods do. We will examine those approaches in the next chapter.
What a Frequency-Driven Tokenizer Does Instead
Now let us see what BPE, the algorithm behind GPT-style tokenizers from Part V, does with these same words. We train a deliberately tiny BPE model (a 250-token vocabulary) on a sample of Melville's Moby Dick. A small vocabulary forces heavy fragmentation, which mimics what happens to morphologically complex languages under production-sized budgets.
from tokenizers import Tokenizer, models, pre_tokenizers, trainers
raw_text = " ".join(gutenberg.raw("melville-moby_dick.txt").split())
with open("moby_train_sample.txt", "w") as f:
f.write(raw_text[:400_000])
bpe = Tokenizer(models.BPE())
bpe.pre_tokenizer = pre_tokenizers.Whitespace()
trainer = trainers.BpeTrainer(vocab_size=250, initial_alphabet=[])
bpe.train(["moby_train_sample.txt"], trainer=trainer)whales -> ['whal', 'es'] swimming -> ['s', 'w', 'im', 'm', 'ing'] unhappiness -> ['un', 'h', 'ap', 'p', 'in', 'ess'] boats -> ['bo', 'at', 's'] harpooner -> ['h', 'ar', 'po', 'on', 'er']
With its vocabulary capped at 250 tokens, the tokenizer has learned whatever substrings happened to be frequent in whaling-era prose, and it fragments everything else into character-level leftovers. The trainer's vocab_size is a cap on the total learned vocabulary, including the initial alphabet and any special tokens; it is not the number of merge operations. Notice that the fragments respect frequency, not meaning: a morpheme like un- survives or shatters depending on how often words beginning with un- appeared in the training text, not on whether it carries a consistent meaning. Scale the vocabulary up to modern sizes and the fit improves, but the principle holds. Subword tokenization approximates morphology statistically, and the approximation degrades precisely where morphology is most productive: rare, novel, and complex words.
Let me quantify that degradation. We compare three groups of words: common bases, mostly regularly inflected or derived forms of those bases, and novel-looking coinages assembled from familiar morphemes that no tokenizer has ever seen.
base_words = [
"whale",
"boat",
"sea",
"ship",
"water",
"captain",
"harpoon",
"ocean",
"crew",
"wind",
]
formed_words = [
"whales",
"boating",
"seaward",
"shipping",
"waterlogged",
"captains",
"harpoons",
"oceanic",
"crewing",
"winds",
]
novel_words = [
"whaleness",
"boatify",
"seawardsman",
"shiphood",
"wateristic",
"captainless",
"harpoonerly",
"oceanite",
"crewdom",
"windageous",
]
groups = {
"Common bases": base_words,
"Formed words": formed_words,
"Novel words": novel_words,
}
avg_subword_counts = {
name: sum(len(bpe.encode(w).tokens) for w in words) / len(words)
for name, words in groups.items()
}
The trend is exactly what morphological theory predicts. Common bases fragment least because they are frequent enough for BPE to have merged them early. Formed words fragment more, and novel words, though built entirely from familiar morphemes, fragment worst of all, since their surface forms never contributed to any merge decision. A human reader decodes captainless instantly from known parts. The tokenizer sees an alien string. This gap between compositional meaning and statistical fragmentation is the core argument for bringing morphological knowledge into tokenization.
The gap should narrow as the vocabulary grows, but it never closes completely. Let us verify this directly by retraining BPE at a range of vocabulary sizes and tracking fragmentation for each group.
vocab_sizes = [150, 250, 500, 1000, 2000, 3000]
frag_by_vocab = {name: [] for name in groups}
for v in vocab_sizes:
tok_v = Tokenizer(models.BPE())
tok_v.pre_tokenizer = pre_tokenizers.Whitespace()
tok_v.train(
["moby_train_sample.txt"],
trainers.BpeTrainer(vocab_size=v, initial_alphabet=[]),
)
for name, words in groups.items():
frag_by_vocab[name].append(
sum(len(tok_v.encode(w).tokens) for w in words) / len(words)
)
@fig-bpe-fragmentation-vs-vocab-size confirms the theoretical prediction: larger vocabulary budgets buy better coverage for everyone, but the ordering of the curves never changes. Statistical tokenization compresses what is frequent; morphology is precisely the machinery for handling what is not.
One Lemma, Many Tokens
To close the loop, let us measure the phenomenon that motivates everything above: how a single lexeme scatters across many surface forms in real text. We count the forms of the verb SAY in Moby Dick, which happens to include both a regular -s/-ing paradigm and the suppletive past tense said.
moby_tokens = [w.lower() for w in gutenberg.words("melville-moby_dick.txt")]
moby_freq = Counter(moby_tokens)
say_family = ["say", "says", "said", "saying"]
family_counts = {form: moby_freq[form] for form in say_family}
family_total = sum(family_counts.values())
distinct_forms = sum(1 for c in family_counts.values() if c > 0)said 304 occurrences (47.9% of family) say 244 occurrences (38.5% of family) says 46 occurrences (7.3% of family) saying 40 occurrences (6.3% of family) 4 distinct surface forms share the lexeme SAY (634 total occurrences)
Here is the practical consequence. An N-gram language model, as built in Part II, learns said and says as unrelated symbols with wildly different statistics, even though they differ only in tense. Every count for said teaches it nothing about says. Multiply this scattering across a whole vocabulary, and inflection becomes a leading source of sparsity, the very problem our smoothing techniques fought to contain. A model with access to morphology, whether through lemmatization before counting or through morphologically informed representations, pools the evidence of all four forms under one lexeme and generalizes from regular patterns to unseen combinations.