Word Tokenization › Contractions and Clitics
Language AI HandbookAll chapters Word Tokenization › Contractions and CliticsOne section · Sections in this chapter5 / 17 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Contractions present a linguistic puzzle. "Don't" is one orthographic word but represents two morphemes: "do" and "not". How should we tokenize it? The answer depends on why you are tokenizing in the first place, and there are principled arguments for each approach.
Understanding why this matters requires thinking about what you want downstream systems to learn. If a language model sees "don't" as a single token, it learns a representation for "don't" as a whole but might not learn that "don't" and "do not" are semantically equivalent. If it sees "do" and "n't" separately, it can potentially generalize: "can n't", "should n't", "would n't" all share the same negation structure. The split form exposes linguistic regularity that the fused form hides.
A clitic is a morpheme that has syntactic characteristics of a word but is phonologically dependent on an adjacent word. English contractions like "'s", "'ll", and "n't" are clitics that attach to their host words. The term comes from Greek "klitikos" meaning "leaning", because clitics lean on adjacent words for their phonological shape. Understanding clitics helps explain why tokenizers split contractions the way they do: we are recovering the syntactic word from its phonologically reduced, attached form.
Different traditions handle contractions differently, and each choice reflects a different theory of what the downstream system needs:
- Keep together: "don't" becomes ["don't"]. Preserves the surface form exactly. Good for tasks where the exact written form matters.
- Split at apostrophe: "don't" becomes ["don", "'t"]. Exposes the components but produces a non-standard split.
- Expand fully: "don't" becomes ["do", "n't"]. Reveals the morphological structure in a linguistically principled way.
- Normalize: "don't" becomes ["do", "not"]. Eliminates contraction variation entirely, which can help frequency counting but loses information about register and style.
# Common English contractions and their expansions
contractions = {
"n't": " n't", # don't -> do n't
"'ll": " 'll", # I'll -> I 'll
"'re": " 're", # we're -> we 're
"'ve": " 've", # I've -> I 've
"'d": " 'd", # I'd -> I 'd
"'m": " 'm", # I'm -> I 'm
"'s": " 's", # it's -> it 's (possessive or "is")
}
def contraction_tokenize(text):
"""Tokenize with contraction splitting."""
# Handle contractions
for contraction, replacement in contractions.items():
text = text.replace(contraction, replacement)
# Standard punctuation handling
text = re.sub(r'([.,!?;:"\(\)\[\]])', r" \1 ", text)
return [t for t in text.split() if t]
# Test contraction handling
contraction_text = "I can't believe she's already gone. We'll miss her."
contraction_tokens = contraction_tokenize(contraction_text)Text: 'I can't believe she's already gone. We'll miss her.' Tokens: ['I', 'ca', "n't", 'believe', 'she', "'s", 'already', 'gone', '.', 'We', "'ll", 'miss', 'her', '.'] Contraction analysis: 'n't' <- clitic (attached to previous word) ''s' <- clitic (attached to previous word) ''ll' <- clitic (attached to previous word)
Splitting contractions reveals the underlying structure. "Can't" becomes "can" and "n't", exposing the negation. "She's" becomes "she" and "'s", which could be either possessive or "is" depending on context. Note that this simple approach differs from Penn Treebank conventions, which split "can't" into "ca" and "n't" to better reflect the morphological structure. The Penn Treebank convention is linguistically motivated: in "can't", the word "can" loses its final 'n' through phonological reduction, so the morpheme boundary is between "ca" and "n't", not "can" and "'t".
The Possessive Problem
The possessive "'s" is particularly tricky. In "John's book", the "'s" marks possession. In "John's running", it's a contraction of "is". Both look identical, and only context reveals the difference. This is a case where tokenization alone cannot fully resolve the ambiguity, and the resolution requires syntactic or semantic understanding that comes later in the pipeline.
The practical implication is that tokenizers must make a choice: either always split "'s" (accepting that the downstream system will need to figure out whether it is possessive or a contraction) or never split it (accepting that possession and contraction are handled identically). Most English tokenizers choose to split, because the downstream benefit of exposing the possessive marker is generally worth it.
# Ambiguous 's examples
ambiguous_examples = [
"John's book is on the table.", # Possessive
"John's running late again.", # Contraction of "is"
"The dog's barking loudly.", # Contraction of "is"
"The dog's tail is wagging.", # Possessive
]
# Tokenize each
ambiguous_tokens = [contraction_tokenize(ex) for ex in ambiguous_examples]Ambiguous 's tokenization: -------------------------------------------------- Text: John's book is on the table. Tokens: ['John', "'s", 'book', 'is', 'on', 'the', 'table', '.'] Interpretation: Possessive Text: John's running late again. Tokens: ['John', "'s", 'running', 'late', 'again', '.'] Interpretation: Contraction Text: The dog's barking loudly. Tokens: ['The', 'dog', "'s", 'barking', 'loudly', '.'] Interpretation: Contraction Text: The dog's tail is wagging. Tokens: ['The', 'dog', "'s", 'tail', 'is', 'wagging', '.'] Interpretation: Possessive
A tokenizer can't distinguish these cases without understanding grammar. This is a fundamental limitation: tokenization is a preprocessing step that happens before syntactic analysis. We split "'s" consistently and leave disambiguation to later stages. This is a reasonable design choice: tokenization should be fast and simple, and it should preserve all the information needed for later stages to do their work. A tokenizer that tried to resolve this ambiguity would be slow, potentially wrong, and would need to be retrained when moving to new domains.