Word Tokenization › Penn Treebank Tokenization
Language AI HandbookAll chapters Word Tokenization › Penn Treebank TokenizationOne section · Sections in this chapter6 / 17 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
The Penn Treebank tokenization standard emerged from the Penn Treebank project, one of the most influential resources in the history of NLP. Completed at the University of Pennsylvania in the early 1990s, the project annotated approximately one million words of Wall Street Journal text with syntactic parse trees. To build consistent parse trees, the annotators needed consistent token boundaries, and the tokenization conventions they developed became the de facto standard for English NLP for the next two decades.
The Penn Treebank standard is significant not because it is the only possible tokenization, or even the best for all tasks, but because it is the standard that most English NLP resources assume. If you are working with a pre-trained language model, a named entity recognizer, or a dependency parser that was trained on Penn Treebank data, then mismatching the tokenization convention can hurt performance significantly. The model learned from a particular kind of input; feeding it differently tokenized text is like giving someone a dictionary in a different dialect.
The key conventions of Penn Treebank tokenization are:
- Split contractions in a linguistically motivated way: "don't" becomes ["do", "n't"]
- Separate all punctuation from words, including periods and commas
- Keep abbreviations together: "Dr." stays as a single token
- Normalize special characters: straight quotes become directional quote tokens
- Normalize brackets: parentheses and square brackets become labeled tokens like -LRB- and -RRB-
from nltk.tokenize import TreebankWordTokenizer
# Initialize the tokenizer
treebank = TreebankWordTokenizer()
# Test on various examples
examples = [
"I can't believe it's not butter!",
"Dr. Smith's patients don't like waiting.",
"The stock rose $5.25 (3.5%) today.",
'"Hello," she said, "how are you?"',
]
treebank_results = [(ex, treebank.tokenize(ex)) for ex in examples]Penn Treebank Tokenization:
============================================================
Text: I can't believe it's not butter!
Tokens: ['I', 'ca', "n't", 'believe', 'it', "'s", 'not', 'butter', '!']
Text: Dr. Smith's patients don't like waiting.
Tokens: ['Dr.', 'Smith', "'s", 'patients', 'do', "n't", 'like', 'waiting', '.']
Text: The stock rose $5.25 (3.5%) today.
Tokens: ['The', 'stock', 'rose', '$', '5.25', '(', '3.5', '%', ')', 'today', '.']
Text: "Hello," she said, "how are you?"
Tokens: ['``', 'Hello', ',', "''", 'she', 'said', ',', '``', 'how', 'are', 'you', '?', "''"]The Treebank tokenizer handles common English word-boundary cases according to established conventions. Notice how it splits "can't" into "ca" and "n't", keeps "Dr." together, and handles the dollar amount and percentage. The splitting of "can't" into "ca" and "n't" rather than "can" and "'t" reflects the historical and morphological fact that "can't" is a contraction of "cannot", not "can not". The morpheme boundary falls after "ca" because "cannot" is a fused compound, not two separate words joined with an apostrophe.
Treebank Conventions
The Penn Treebank standard makes specific choices that were designed to support syntactic annotation and parsing. Each convention has a rationale rooted in the needs of the original project.
| Input | Output | Rationale |
|---|---|---|
| don't | do n't | Reveals negation structure |
| I'm | I 'm | Separates subject from verb |
| they're | they 're | Consistent clitic handling |
| John's | John 's | Possessive/contraction split |
| "Hello" | `` Hello '' | Directional quotes |
| (test) | -LRB- test -RRB- | Bracket normalization |
# Demonstrate specific Treebank conventions
conventions = [
("Contractions", "I don't think they're coming."),
("Possessives", "John's car and Mary's bike."),
("Quotes", '"Hello," she said.'),
("Brackets", "The result (see Figure 1) is clear."),
("Currency", "It costs $19.99 plus 8% tax."),
]
convention_results = [
(name, text, treebank.tokenize(text)) for name, text in conventions
]Treebank Convention Examples:
------------------------------------------------------------
Contractions:
Input: I don't think they're coming.
Output: ['I', 'do', "n't", 'think', 'they', "'re", 'coming', '.']
Possessives:
Input: John's car and Mary's bike.
Output: ['John', "'s", 'car', 'and', 'Mary', "'s", 'bike', '.']
Quotes:
Input: "Hello," she said.
Output: ['``', 'Hello', ',', "''", 'she', 'said', '.']
Brackets:
Input: The result (see Figure 1) is clear.
Output: ['The', 'result', '(', 'see', 'Figure', '1', ')', 'is', 'clear', '.']
Currency:
Input: It costs $19.99 plus 8% tax.
Output: ['It', 'costs', '$', '19.99', 'plus', '8', '%', 'tax', '.']The Treebank tokenizer converts ASCII quotes to directional quote tokens (`` and '') and brackets to labeled tokens (-LRB- and -RRB-). These conventions normalize text for consistent downstream processing. The bracket normalization is particularly important for parsing: if parentheses were left as raw characters, a parser might confuse them with syntactic structures, since parse trees are often written in parenthesized notation.
Understanding these conventions matters when you work with pre-trained NLP models. If a named entity recognizer or a dependency parser was trained on Penn Treebank data, and you feed it text tokenized with a different convention, the mismatch can degrade performance. The model learned to expect "ca n't" as two separate tokens; giving it "can't" as one token produces a token it has never seen before.