Word Tokenization › Worked Example: Tokenizing a Realistic Document

Michael BrenndoerferUpdated March 21, 20253 min read
Language AI HandbookAll chapters
Part 1 of 67Text as Data
Chapter 5Word Tokenization › Worked Example: Tokenizing a Realistic Document
Reading section 10 of 17 · Worked Example: Tokenizing a Realistic Document · 3 minSection 10 of 17 · 3 min

Book contents

Language AI Handbook67 parts · 513 chaptersView the whole book →
Active chapter 5 of 513

Section 10 of 17 · 3 min read

Word Tokenization › Worked Example: Tokenizing a Realistic DocumentOne section · Sections in this chapter10 / 17 ↓
Sections in this chapter17
Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Let's trace through a realistic tokenization scenario step by step, using a short paragraph that contains several of the challenges we have discussed. This worked example will make the sequence of decisions concrete.

Consider this text: "Prof. O'Brien's 2024 study found that 3.5% of patients said they'd improved. (See Table 1.) The drug 'X-27' showed promising results."

Let's trace what a Penn Treebank-style tokenizer would do with this text:

  1. Identify and protect abbreviations: "Prof." is an abbreviation, so its period is protected. The tokenizer marks it as PROF internally to prevent the period from being separated.

  2. Handle contractions: "they'd" contains the contraction "'d", so it becomes "they 'd". The apostrophe is the split point.

  3. Handle possessive apostrophes: "O'Brien's" contains "'s". The tokenizer splits this into "O'Brien" and "'s". Note that "O'Brien" itself contains an apostrophe as part of the name, which the tokenizer must preserve.

  4. Separate punctuation: Separate punctuation marks from adjacent words. "improved." becomes "improved" and ".". The parentheses around "(See Table 1.)" become separate -LRB- and -RRB- tokens.

  5. Handle numbers and percentages: "3.5%" stays together as a number, though the "%" sign would typically be separated.

  6. Clean up and split: After all transformations, the text is split on whitespace and empty tokens are removed.

The final token sequence would be something like: ["Prof.", "O'Brien", "'s", "2024", "study", "found", "that", "3.5", "%", "of", "patients", "said", "they", "'d", "improved", ".", "-LRB-", "See", "Table", "1", ".", "-RRB-", "The", "drug", "'X-27'", "showed", "promising", "results", "."]

This trace reveals several interesting edge cases. "O'Brien" has an apostrophe that is part of the name, not a contraction, so the tokenizer must not split it. "X-27" is a product name with a hyphen, which the tokenizer keeps together. "2024" is a year, treated as a single token. "3.5%" involves both a decimal point and a percent sign.

In[42]:
Code
# The worked example sentence
worked_example = "Prof. O'Brien's 2024 study found that 3.5% of patients said they'd improved. (See Table 1.) The drug 'X-27' showed promising results."

# Tokenize with NLTK word_tokenize (Penn Treebank style)
worked_tokens = word_tokenize(worked_example)
Out[43]:
Console
Worked Example:
Input: Prof. O'Brien's 2024 study found that 3.5% of patients said they'd improved. (See Table 1.) The drug 'X-27' showed promising results.

Tokens (31):
  [ 0] 'Prof.'
  [ 1] 'O'Brien'
  [ 2] ''s'
  [ 3] '2024'
  [ 4] 'study'
  [ 5] 'found'
  [ 6] 'that'
  [ 7] '3.5'
  [ 8] '%'
  [ 9] 'of'
  [10] 'patients'
  [11] 'said'
  [12] 'they'
  [13] ''d'
  [14] 'improved'
  [15] '.'
  [16] '('
  [17] 'See'
  [18] 'Table'
  [19] '1'
  [20] '.'
  [21] ')'
  [22] 'The'
  [23] 'drug'
  [24] '''
  [25] 'X-27'
  [26] '''
  [27] 'showed'
  [28] 'promising'
  [29] 'results'
  [30] '.'

Looking at the output, you can verify each of the steps we traced above. The tokenizer has made specific decisions about each ambiguous case, and the resulting tokens are ready for downstream processing. Notice that the apostrophe in "O'Brien" is preserved, "3.5" and "%" are separated, and the parentheses become their own tokens. These decisions reflect the accumulated wisdom of the Penn Treebank project about what makes tokenization useful for NLP.