Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Regular Expressions › Building a Text Cleaning Pipeline

Michael BrenndoerferUpdated March 18, 20252 min read

Language AI HandbookAll chapters

Part 1 of 67Text as Data

Chapter 3Regular Expressions › Building a Text Cleaning Pipeline

Reading section 15 of 21 · Building a Text Cleaning Pipeline · 2 minSection 15 of 21 · 2 min

Section 15 of 21 · 2 min read

Regular Expressions › Building a Text Cleaning PipelineOne section · Sections in this chapter15 / 21 ↓

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Let's combine what we've learned into a practical text preprocessing pipeline. A cleaning pipeline is a sequence of regex operations applied in a specific order. Order matters: removing URLs before tokenizing prevents partial URL fragments from becoming spurious tokens. Normalizing whitespace after all substitutions ensures that multiple removals do not leave gaps with multiple spaces. The design principle is to always think about the pipeline as a sequence of transformations that progressively bring the text closer to the clean, normalized form your downstream model expects.

In[67]:

Code

import re
from typing import List, Tuple


class TextCleaner:
    """A regex-based text cleaning pipeline for NLP preprocessing."""

    def __init__(self):
        # Compile patterns once
        self.url_pattern = re.compile(r"https?://\S+|www\.\S+")
        self.email_pattern = re.compile(r"\S+@\S+\.\S+")
        self.mention_pattern = re.compile(r"@\w+")
        self.hashtag_pattern = re.compile(r"#\w+")
        self.number_pattern = re.compile(r"\b\d+(?:\.\d+)?\b")
        self.whitespace_pattern = re.compile(r"\s+")
        self.punctuation_pattern = re.compile(r"[^\w\s]")

    def remove_urls(self, text: str) -> str:
        return self.url_pattern.sub(" ", text)

    def remove_emails(self, text: str) -> str:
        return self.email_pattern.sub(" ", text)

    def remove_mentions(self, text: str) -> str:
        return self.mention_pattern.sub(" ", text)

    def extract_hashtags(self, text: str) -> Tuple[str, List[str]]:
        hashtags = self.hashtag_pattern.findall(text)
        cleaned = self.hashtag_pattern.sub(" ", text)
        return cleaned, hashtags

    def normalize_whitespace(self, text: str) -> str:
        return self.whitespace_pattern.sub(" ", text).strip()

    def clean(self, text: str) -> str:
        """Apply full cleaning pipeline."""
        text = self.remove_urls(text)
        text = self.remove_emails(text)
        text = self.remove_mentions(text)
        text, _ = self.extract_hashtags(text)
        text = self.normalize_whitespace(text)
        return text


# Test the pipeline
cleaner = TextCleaner()

sample = """
Check out https://example.com for more info! 
Contact support@company.com or @helpdesk
#MachineLearning is amazing! #NLP #AI
"""

cleaned = cleaner.clean(sample)
_, hashtags = cleaner.extract_hashtags(sample)

Out[68]:

Console

Original text:

Check out https://example.com for more info! 
Contact support@company.com or @helpdesk
#MachineLearning is amazing! #NLP #AI


Cleaned text:
  'Check out for more info! Contact or is amazing!'

Extracted hashtags:
  ['#MachineLearning', '#NLP', '#AI']