Word Tokenization › Using NLTK and spaCy
Language AI HandbookAll chapters Word Tokenization › Using NLTK and spaCyOne section · Sections in this chapter9 / 17 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Production NLP work typically uses established libraries rather than custom tokenizers. NLTK and spaCy are the two most popular choices for English, and each reflects a different design philosophy. NLTK provides a collection of tokenizers with different characteristics, giving you flexibility to choose the one that matches your conventions. spaCy provides a single, highly optimized tokenizer with rich linguistic annotations.
The choice between building your own tokenizer and using a library is similar to the choice between writing your own sorting algorithm and using the standard library's sort. For learning and understanding, writing your own is invaluable. For production work, the library version is almost always better: it has been tested on billions of words, handles edge cases you haven't thought of, is faster than anything you'll write yourself, and comes with documentation and community support.
NLTK Tokenizers
NLTK provides several tokenization options, each implementing a different approach. Understanding which tokenizer to use requires understanding what each one does:
from nltk.tokenize import (
TreebankWordTokenizer,
word_tokenize,
wordpunct_tokenize,
)
text = "Dr. Smith's patients don't like waiting. It's 3:30pm!"
# Different NLTK tokenizers
nltk_word = word_tokenize(text)
nltk_wordpunct = wordpunct_tokenize(text)
nltk_treebank = TreebankWordTokenizer().tokenize(text)Text: Dr. Smith's patients don't like waiting. It's 3:30pm! NLTK word_tokenize: ['Dr.', 'Smith', "'s", 'patients', 'do', "n't", 'like', 'waiting', '.', 'It', "'s", '3:30pm', '!'] NLTK wordpunct_tokenize: ['Dr', '.', 'Smith', "'", 's', 'patients', 'don', "'", 't', 'like', 'waiting', '.', 'It', "'", 's', '3', ':', '30pm', '!'] NLTK TreebankWordTokenizer: ['Dr.', 'Smith', "'s", 'patients', 'do', "n't", 'like', 'waiting.', 'It', "'s", '3:30pm', '!']
Each tokenizer makes different choices. word_tokenize uses the Punkt sentence tokenizer combined with a word tokenizer and generally follows Penn Treebank conventions. wordpunct_tokenize splits on both whitespace and punctuation characters, which means it splits "Dr." into "Dr" and ".", treating all punctuation as delimiters without exception. TreebankWordTokenizer follows Penn Treebank conventions strictly, including the normalization of quotes and brackets.
The word_tokenize function is the most commonly used NLTK tokenizer and the best default choice for English text. It is trained on real corpus data and handles most common cases correctly. If you are unsure which NLTK tokenizer to use, start with word_tokenize.
spaCy Tokenizer
spaCy's tokenizer is rule-based but highly optimized and configurable. Unlike NLTK, which offers multiple tokenizers for different situations, spaCy provides a single tokenizer that is designed to be the right choice for almost all English NLP tasks. The tokenizer is also the first step in spaCy's full processing pipeline, which adds part-of-speech tags, dependency parses, and named entity labels on top of the tokens.
The key design principle of spaCy's tokenizer is that it is deterministic and consistent. Given the same input, it always produces the same output, which is critical for reproducibility. It also integrates with the rest of spaCy's pipeline, so that tokenization and linguistic annotation happen in a single pass over the text.
import spacy
# Load English model when installed; otherwise use spaCy's tokenizer-only fallback.
try:
nlp = spacy.load("en_core_web_sm")
except OSError:
nlp = spacy.blank("en")
# Tokenize
doc = nlp(text)
spacy_tokens = [token.text for token in doc]
# spaCy also provides additional information
token_info = [
(token.text, token.pos_, token.is_punct, token.is_stop) for token in doc
]Text: Dr. Smith's patients don't like waiting. It's 3:30pm! spaCy tokens: ['Dr.', 'Smith', "'s", 'patients', 'do', "n't", 'like', 'waiting', '.', 'It', "'s", '3:30pm', '!'] Token details: Token POS Punct? Stop? ---------------------------------------- Dr. PROPN False False Smith PROPN False False 's PART False True patients NOUN False False do AUX False True n't PART False True like VERB False False waiting VERB False False . PUNCT True False It PRON False True 's AUX False True 3:30pm NUM False False ! PUNCT True False
spaCy provides tokens along with part-of-speech tags, punctuation flags, and stop word indicators. This additional information is useful for downstream processing. The fact that spaCy gives you POS tags, punctuation flags, stop word flags, and lemmas as part of tokenization means that you can do quite a lot of text analysis using only the output of the tokenizer, without needing to run additional models.
Comparing Tokenizers
Let's compare how different tokenizers handle the same challenging text. This comparison illustrates that tokenization is a design choice, not a solved problem with a single correct answer:
challenge_text = (
"I can't believe she's already gone! We'll miss her. #sad @friends"
)
# Tokenize with different tools
results = {
"Whitespace": challenge_text.split(),
"NLTK word_tokenize": word_tokenize(challenge_text),
"NLTK Treebank": TreebankWordTokenizer().tokenize(challenge_text),
"spaCy": [t.text for t in nlp(challenge_text)],
"Our tokenizer": tokenizer.tokenize(challenge_text),
}Text: I can't believe she's already gone! We'll miss her. #sad @friends Tokenizer Comparison: ============================================================ Whitespace: ['I', "can't", 'believe', "she's", 'already', 'gone!', "We'll", 'miss', 'her.', '#sad', '@friends'] Count: 11 NLTK word_tokenize: ['I', 'ca', "n't", 'believe', 'she', "'s", 'already', 'gone', '!', 'We', "'ll", 'miss', 'her', '.', '#', 'sad', '@', 'friends'] Count: 18 NLTK Treebank: ['I', 'ca', "n't", 'believe', 'she', "'s", 'already', 'gone', '!', 'We', "'ll", 'miss', 'her.', '#', 'sad', '@', 'friends'] Count: 17 spaCy: ['I', 'ca', "n't", 'believe', 'she', "'s", 'already', 'gone', '!', 'We', "'ll", 'miss', 'her', '.', '#', 'sad', '@friends'] Count: 17 Our tokenizer: ['I', 'ca', 'n', "'", 't', 'believe', 'she', "'", 's', 'already', 'gone', '!', 'We', "'", 'll', 'miss', 'her', '.', '#sad', '@friends'] Count: 20
Different tokenizers make different choices about contractions and punctuation; they also differ in how they handle social-media elements such as hashtags and mentions. There's no universally "correct" answer. The best choice depends on your application.
The following table summarizes how each tokenizer performs on the challenge text:
| Tokenizer | Token Count | Key Characteristics |
|---|---|---|
| Whitespace | 9 | Keeps punctuation attached, fewest tokens |
| NLTK word_tokenize | 17 | Splits contractions, separates punctuation |
| NLTK Treebank | 17 | Penn Treebank conventions, normalizes quotes |
| spaCy | 17 | Rich token attributes (POS, flags) |
| Our tokenizer | 17 | Custom rules, configurable behavior |
The variation in token counts (from 9 to 17 for the same input) highlights that tokenization is not a solved problem with a single correct answer. Whitespace tokenization produces the fewest tokens by keeping punctuation attached, while linguistic tokenizers like NLTK and spaCy produce nearly twice as many tokens by separating punctuation and splitting contractions. For production work, any of the linguistic tokenizers is a reasonable choice, and the differences between them are usually small enough that they don't significantly affect downstream performance.