Text Preprocessing: Tokenization › Core Preprocessing Techniques

Imagine you're teaching a computer to read. Unlike humans who instantly recognize words, sentences, and meaning, computers see text as nothing more than a sequence of characters. The sentence "The cat's sitting on the mat" appears to a computer as 30 characters: letters, spaces, an apostrophe, and a period. Before any machine learning model can understand this text, we must transform it from a continuous stream of characters into discrete, meaningful units that algorithms can process.

This transformation is the essence of text preprocessing, and it requires answering three fundamental questions:

1. How do we break text into meaningful units? Where do words begin and end? How do we handle punctuation, contractions, and special cases?
2. How do we handle variation? "Running," "running," and "RUNNING" represent the same concept but appear as different strings. How do we standardize these variations?
3. What should we remove? Real-world text contains noise: HTML tags, URLs, typos, and formatting artifacts. What information is essential, and what can we safely discard?

These questions lead us to three core preprocessing operations: tokenization breaks text into discrete units, normalization standardizes variations, and cleaning removes noise. But understanding these techniques requires more than memorizing definitions. We need to grasp why each operation is necessary and how they work together to transform messy human language into structured data that models can learn from.

Let's build this understanding step by step, starting with the most fundamental question: how do we identify where one word ends and another begins?

Tokenization answers our first fundamental question: how do we break continuous text into discrete, meaningful units? This is the foundation of all text processing because you cannot normalize, clean, or analyze text until you've identified its basic building blocks.

Think of tokenization like cutting a loaf of bread into slices. The loaf is the continuous text string, and each slice is a token, a discrete unit we can work with. But here's the challenge: unlike bread, text doesn't have natural "cut lines." Where exactly should we split "don't"? Is it one word or two? Should "U.S.A." be one token or three? These ambiguities force us to make decisions that affect everything downstream.

Why tokenization matters: Without tokenization, we can't count word frequencies, build vocabularies, or apply any statistical or machine learning techniques. A model that sees "The cat's sitting" as 30 characters has no way to learn that "cat" and "sitting" are meaningful units. Tokenization transforms the problem from "understanding 30 characters" to "understanding 7 tokens," dramatically reducing complexity.

The challenge of tokenization lies in ambiguity. Spaces seem like obvious delimiters in English, but they're insufficient for handling:

• Contractions like "don't" (one word or two?)
• Hyphenated words like "state-of-the-art" (one word or four?)
• Punctuation: should "mat." be "mat" + "." or just "mat."?
• Special cases: URLs, email addresses, abbreviations

Different languages compound these challenges: Chinese and Japanese don't use spaces between words, German creates long compound words, and Arabic uses diacritical marks that may or may not be present.

The progression of tokenization methods: We'll explore tokenization by starting with the simplest approach and progressively building more sophisticated methods. Each method solves some problems while introducing new tradeoffs, teaching us what's truly necessary for robust text processing.

Whitespace Tokenization

The most naive approach simply splits text on whitespace characters (spaces, tabs, newlines). This works surprisingly well for English text that has already been cleaned, but fails on several common cases:

The key insight is that morphological patterns (prefixes like "un-", suffixes like "-able") appear across many words. By learning these subword units from data, models can understand rare or novel words by composing their parts. This is how BERT and GPT handle words they've never seen before.

Once we've tokenized our text, we've solved the problem of identifying word boundaries. But we've created a new problem: vocabulary explosion. Consider what happens when we tokenize a simple sentence:

• "The cat runs" → ["The", "cat", "runs"]
• "The cat ran" → ["The", "cat", "ran"]
• "The CAT RUNS" → ["The", "CAT", "RUNS"]

To a computer, these are completely different tokens. "runs," "ran," and "RUNS" are treated as three distinct words, even though they represent the same concept. This variation creates several problems:

1. Vocabulary size explosion: Instead of learning one representation for "run," a model must learn separate representations for "run," "runs," "ran," "running," "RUN," "RUNS," etc. This wastes model capacity and training data.

2. Data sparsity: With more unique tokens, each token appears less frequently. Rare tokens have unreliable statistics, making it harder for models to learn meaningful patterns.

3. Generalization failure: A model trained on "running" might not recognize "runs" as related, even though they're linguistically connected.

Normalization solves this by mapping variations to canonical forms. Instead of treating "running," "runs," and "RUNNING" as different tokens, normalization reduces them to a common representation. This decreases vocabulary size, increases token frequency, and helps models recognize that morphologically related words share meaning.

But normalization is a tradeoff: we gain efficiency and generalization at the cost of losing information. "EXCITED!!!" conveys more emotion than "excited," and "Apple" (the company) differs from "apple" (the fruit). Understanding when to normalize and how aggressively is crucial for building effective NLP systems.

Case Normalization

The simplest normalization converts all text to lowercase. This reduces vocabulary size by treating "The," "the," and "THE" as identical:

Lemmatization correctly handles irregular verbs like "ran" → "run" and "was" → "be," but requires knowing the part-of-speech (verb, noun, adjective) for accurate results. This makes it more expensive but linguistically sound.

When to Normalize

The tradeoff between stemming and lemmatization mirrors a broader question: how much normalization is appropriate? Aggressive normalization reduces vocabulary size and improves generalization but loses information. For small datasets and simple models like Naive Bayes, aggressive normalization helps by reducing sparsity. For large datasets and neural models, minimal normalization often works better because models can learn morphological patterns from data.

Modern pretrained transformers like BERT often use subword tokenization instead of stemming or lemmatization. Their WordPiece vocabulary naturally handles morphology: "running" might tokenize as ["run", "##ning"], allowing the model to learn that words ending in "##ning" relate to their base form.

After tokenization and normalization, we have discrete, standardized tokens. But real-world text contains more than just words. It's filled with noise: formatting artifacts, non-linguistic content, and encoding inconsistencies that don't contribute to meaning but can confuse models and waste computational resources.

Why cleaning matters: Consider what happens when we process web-scraped text. A sentence like "Check out this site: https://example.com!" contains:

• HTML tags (<b>, </b>) that are formatting instructions, not linguistic content
• A URL that's a reference, not a word
• Punctuation that may or may not be meaningful

If we don't clean this text, our tokenizer might create tokens like <b>, </b>, https://example.com, which:

• Add noise to our vocabulary without contributing semantic meaning
• Waste model capacity learning patterns in formatting artifacts
• Create spurious correlations (e.g., learning that <b> appears near certain words)

The cleaning challenge: Cleaning requires deciding what's signal versus noise, and this decision is task-dependent. For sentiment analysis, emojis and punctuation intensity markers (like "!!!") are crucial signals. For document classification, they might be noise. For medical text, numbers are essential; for topic modeling, they might be irrelevant.

Cleaning techniques remove:

• Formatting artifacts: HTML tags, markdown, special characters
• Non-linguistic content: URLs, email addresses, phone numbers (depending on task)
• Encoding inconsistencies: Unicode normalization, character encoding issues
• High-frequency noise: Stopwords (words like "the," "is" that appear everywhere but carry little semantic information for many tasks)

The key insight is that cleaning is about selective removal. We remove what doesn't help our specific task while preserving what does. This requires understanding both your data and your downstream application.

Stopword Removal

Stopwords are high-frequency words like "the," "is," "at," and "which" that occur in nearly every document but carry little semantic information for many tasks. Removing them reduces feature space dramatically: