Regular Expressions for NLP: Pattern Matching in Python

Text data is messy. Emails hide in paragraphs, phone numbers appear in a dozen formats, and dates refuse to follow any single convention. Before you can extract meaning from text, you need to find patterns within it. Regular expressions give you a powerful, compact language for describing these patterns. A single regex can match thousands of variations of an email address, validate input formats, or extract structured data from unstructured text.

Think of a regular expression as a query language for text. Just as SQL lets you describe which rows to retrieve from a database, regex lets you describe which sequences of characters to find in a string. The description is itself written as a string, using a vocabulary of special characters with specific meanings. Once you know the vocabulary, you can compose patterns of enormous expressive power from a handful of building blocks.

The practical impact of regex on NLP pipelines is hard to overstate. Before you feed text to a machine learning model, you typically need to clean it: strip out URLs, anonymize email addresses, normalize whitespace, identify sentence boundaries, and extract structured entities like dates and phone numbers. Each of these tasks can be expressed as one or more regex patterns. A preprocessing pipeline that would take hundreds of lines of imperative Python code can often be reduced to a few well-crafted patterns. The key insight is that text structure tends to be systematic even when its content is wildly variable. An email address always has a local part, an @ symbol, a domain, and a top-level domain extension. A date always has numeric components separated by specific delimiters. Regex captures those structural regularities in a form that is both precise and concise.

Regex is also famously difficult to read. The pattern (?P<year>\d{4})-(?P<month>\d{2})-(?P<day>\d{2}) is meaningful and powerful once you know the syntax, but it looks like line noise to the uninitiated. This chapter demystifies that syntax systematically. Each feature is introduced with motivation and simple examples before more complex patterns are built. By the end, you will understand what each component does and be able to write and maintain patterns confidently.

Regex bridges theoretical computer science and practical software engineering. The patterns you write are evaluated by a finite automaton built from your pattern, connecting everyday text processing to decades of formal language theory. We will touch on that theory briefly so you understand why some patterns are dangerous and why nested quantifiers can bring a server to its knees. But the primary focus is practical: reading and writing regex patterns for real NLP tasks.

This chapter teaches you to read and write regular expressions fluently. You'll learn the syntax that makes regex both powerful and cryptic, understand when to use them versus simpler alternatives, and build practical patterns for common NLP tasks. By the end, you'll wield regex as a precision tool for text manipulation.

A regular expression (regex) is a sequence of characters that defines a search pattern. Think of it as a tiny programming language embedded within Python, specialized for matching and manipulating text. Regex is not unique to Python. The same core syntax works in Java and JavaScript. It also works in Ruby and Perl, in Go, and in dozens of other languages, as well as in command-line tools like grep, sed, and awk. Learning regex once gives you a superpower that transfers across your entire toolkit.

The power of regex comes from its expressiveness. The pattern \b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b looks intimidating, but it matches most email addresses in a single line. Without regex, you'd need dozens of lines of conditional logic to achieve the same result. Consider what that email pattern says: match a word boundary, then one or more characters from the set of letters, digits, dots, underscores, percent signs, plus signs, and hyphens; then a literal @; then one or more characters from letters, digits, dots, and hyphens; then a literal dot; then two or more letters; then another word boundary. Stated in English it takes a full sentence. In regex it takes 48 characters. Notice that the pattern also naturally rejects invalid emails like @missing.com (no local part) or nodomain@ (no domain) without any explicit rejection logic. The structure of the pattern itself defines what is and is not an email address.

Let's start with a simple example:

The regex found "cat" as a standalone word and "cat" inside "catalog". This illustrates a key point: by default, regex matches anywhere in the text, including inside other words. We'll learn how to match whole words only using word boundaries.

Python's re module provides the interface for working with regular expressions. The module ships with Python's standard library, so no installation is needed. It exposes a small set of functions that cover the vast majority of text matching and manipulation tasks. Before diving into pattern syntax, let's understand the main functions you'll use:

Each function serves a different purpose. Use search() when you only need the first match, findall() when you want a simple list of matched strings, finditer() when you need position information or groups, sub() for replacements, and split() to break text at pattern boundaries.

In practice, findall() is the workhorse for NLP preprocessing because you usually want every occurrence of an entity. finditer() becomes needed when you need to know where matches appear, for example to build a token span list or to highlight matches in a UI. sub() is inuseful for cleaning: removing URLs, anonymizing emails, or normalizing whitespace all come down to substitution. all these functions share the same pattern syntax. Learning the syntax once, you can apply it through any of these interfaces.

Notice the r prefix before pattern strings: r'\w+@\w+'. This creates a raw string where backslashes are treated literally. Without it, Python interprets backslashes as escape sequences before the regex engine sees them.

Always use raw strings for regex patterns. It's a habit that will save you from subtle bugs. The rule is simple: every regex pattern string should start with r. The compiler cost is zero and the defensive benefit is high.

Regular expressions use special characters called metacharacters to represent patterns. These characters have meaning beyond their literal value. While literal characters like a or 5 match themselves, metacharacters like ., *, and [] define rules for what to match. Mastering these building blocks is the key to writing effective patterns.

The complete set of regex metacharacters is surprisingly small: . ^ $ * + ? { } [ ] \ | ( ). Every complex pattern you encounter is constructed entirely from these characters combined with literals. This is what makes regex a learnable skill rather than an arbitrary collection of incantations. Once you understand what each metacharacter does, you can read any regex pattern by decomposing it into its components, and you can write new patterns by composing those same components.

Think of metacharacters as the grammar of the regex language. Just as English grammar rules tell you how to combine words into meaningful sentences, metacharacter rules tell you how to combine pattern elements into meaningful descriptions of text structure. The grammar is small but recursive: you can nest groups inside quantifiers, combine character classes with anchors, and chain lookaheads with capturing groups. This composability is what gives regex its power.

The dot . matches any single character except a newline:

The dot matched 'a', 'o', 'u', '@', and '9', but not the newline. To match newlines too, use the re.DOTALL flag or the pattern [\s\S]. The dot is deceptively broad. In practice, using . in a pattern often causes unintended matches because it accepts characters you didn't think about. If you mean "any letter" you should write [a-zA-Z], not .. Reserve . for situations where you mean "any character," such as matching a fixed-width field of unknown content.

Square brackets define a character class, matching any single character from the set. Character classes are one of the most important and frequently used features of regex, because real text rarely has a single accepted character in a given position. An American English name could start with any uppercase letter. A Unix username might contain letters, digits, hyphens, or underscores. A decimal number has digits in all positions. Character classes let you specify exactly which characters are acceptable at each position.

A caret ^ at the start of a character class negates it, matching any character NOT in the set:

Regex provides convenient shortcuts for common character classes:

The following table summarizes the most common regex character classes:

Note that uppercase versions match the complement (negation) of their lowercase counterparts.

The shorthand character classes deserve particular attention in NLP work. The \w class (word characters: letters, digits, and underscore) is a natural building block for matching tokens, but it has an important nuance: it matches the underscore character. This means \w+ matches hello_world as a single token, which may or may not be what you want depending on whether shows appear in your identifiers. The \s class matches all standard whitespace including newlines, which is useful when normalizing multi-line text into single-line tokens.

Quantifiers specify how many times the preceding element should match. They range from simple repetition (*, +, ?) to precise bounds ({n}, {n,m}). Understanding quantifiers is essential because they determine whether your pattern matches once, multiple times, or not at all.

Every quantifier attaches to the immediately preceding element, which can be a literal character, a character class, a shorthand class, or a group enclosed in parentheses. This is a common source of confusion: in ab+, the + applies only to b, so the pattern matches "ab", "abb", "abbb", and so on, but not "aab". If you want to repeat "ab" as a unit, you need (ab)+. Understanding what element a quantifier is attached to is the first step to reading any complex pattern correctly.

The visualization shows how quantifiers dramatically affect matching behavior. Notice that ba* finds 5 matches because it accepts zero 'a's (matching just 'b'), while ba+ finds only 4 because it requires at least one 'a'. The bounded quantifiers {2}, {2,3}, and {2,} are more selective, matching only strings with specific repetition counts.

By default, quantifiers are greedy: they match as much as possible. Adding ? after a quantifier makes it lazy, matching as little as possible.

Anchors match positions in the string rather than characters. Unlike metacharacters that consume text, anchors assert that the current position in the string meets certain criteria. This makes them essential for matching patterns at specific locations, such as the beginning of a line or at word boundaries.

Think of anchors as conditions that must be satisfied at a specific point in the string, not as characters to match. The anchor ^ says "the engine must be at the start of the string right now." The anchor \b says "the engine must be at a word boundary right now." If the condition is not met at the current position, the match fails at that position and the engine tries the next one. Because anchors consume no characters, you can stack them: ^$ matches an empty line (must simultaneously be at the start and end of a line). This zero-width property is what distinguishes anchors from all other regex elements.

The word boundary anchor \b deserves special attention for NLP work. It matches the invisible boundary between a word character and a non-word character. At the beginning of the string, \b matches if the first character is a word character. At the end of the string, \b matches if the last character is a word character. In the middle of a string, \b matches at every transition between \w and \W. This makes \bword\b the correct way to match whole words, because it rejects partial matches like "password" when you only want "word". Without word boundaries, searching for common words like "the" or "at" produces enormous numbers of false positives inside longer words.

Word boundaries are essential for matching whole words. The \b anchor matches the position between a word character and a non-word character. In "catalog", there's no word boundary before or after "cat", so \bcat\b doesn't match it.

In practice, you should default to using word boundaries whenever you search for words in natural text. The cost is negligible and the benefit is avoiding spurious matches. A search for \brun\b correctly identifies the verb in "I run every day" without matching "runtime", "running", "outrun", or "rundown". This is particularly important in NLP pipelines where false positives silently corrupt downstream analysis.

Parentheses serve two purposes in regex: grouping elements together and capturing matched text for later use. Grouping lets you apply quantifiers to multi-character sequences or create alternations. Capturing stores the matched text so you can reference it later in the pattern or in replacement strings.

The distinction between these two uses is important. When you write (https?)://, the parentheses group http and s? so the alternation is clear, and they also capture the matched protocol as group 1. If you do not need the captured content, you should use a non-capturing group (?:https?):// instead. Using capturing groups unnecessarily has two costs: it slightly slows down the match because the engine must track the captured span, and it pollutes the results of findall() with content you do not want.

When you use findall() with groups, it returns only the captured group contents, not the full match. This is often what you want when extracting specific parts of a pattern. The behavior change of findall() based on whether groups are present is a common source of confusion for new regex users. If your pattern contains no groups, findall() returns a list of the full matched strings. If it contains one group, it returns a list of the group's content. If it contains multiple groups, it returns a list of tuples. Understanding this contract up front will save significant debugging time.

Named groups make patterns more readable and self-documenting:

Named groups are especially valuable in complex patterns where numbered groups become confusing. Consider a pattern for parsing log lines that has a dozen groups: numbering them and remembering which number corresponds to which field is error-prone. Named groups make the pattern self-documenting. The match.groupdict() method returns a dictionary mapping each group name to its matched content, which you can pass directly into a data processing function without any manual unpacking.

In practice, teams that maintain regex-heavy code often adopt a convention of always using named groups in production patterns, reserving numbered groups only for throwaway scripts. This makes code review much easier, because reviewers can understand what a substitution like r"\g<year>-\g<month>-\g<day>" is doing without reverse-engineering the pattern.

Sometimes you need grouping for structure but don't want to capture the content. Use (?:...):

Non-capturing groups keep your results clean when you only care about specific parts of the pattern. Use the rule: if you do not intend to reference the group's content, make it non-capturing.

Backreferences let you match the same text that was captured by an earlier group. This is a feature that goes beyond simple pattern matching: it introduces a form of memory into the pattern. Without backreferences, regex can only describe the structure of what to match. With backreferences, it can express constraints across different positions in the string, specifically that two positions must contain the same text.

The \1 refers back to whatever was captured by the first group. In the repeated words example, if the first group captures "the", then \1 only matches another "the", not any word.

Backreferences are powerful for validation tasks. The HTML tag example demonstrates a check that a simple <\w+>.*?</\w+> pattern cannot do: verifying that the closing tag matches the opening tag. This kind of constraint is impossible to express with pure character matching. In practice, backreferences are most useful for finding structural duplications in text, such as repeated words in proofreading tools or mismatched markup in document processing. Keep in mind that backreferences make the regex engine's job harder, because it must track and compare captured content at each position. Patterns with many backreferences can become slow on long documents.

Lookahead and lookbehind assertions match a position based on what comes before or after, without consuming any characters. These are called "zero-width assertions" because they check conditions without advancing the regex engine's position in the string. This makes them powerful for extracting text that appears in specific contexts while excluding the context itself from the match.

Think of lookarounds as asking the question "is the text around my target in the right form?" before returning the target. A positive lookahead (?=...) asks "is this text followed by ...?" A negative lookahead (?!...) asks "is this text NOT followed by ...?" The lookbehind variants ask the same questions about what precedes the target. The key detail is that the lookaround's content is checked but not consumed: it does not become part of the match and does not move the engine forward.

Regex flags modify how patterns are interpreted. They are the equivalent of options or settings that change the rules the engine uses when evaluating a pattern. You can pass flags as the third argument to any re function, or combine multiple flags with the | operator. You can also embed flags directly inside a pattern using inline syntax like (?i) for case-insensitive, which is useful when you are storing patterns as strings and cannot pass them as function arguments.

The re.VERBOSE flag is particularly valuable for complex patterns. It lets you break patterns across lines and add comments, making them maintainable. Consider using re.VERBOSE as the default for any pattern longer than about 20 characters. A pattern that takes five minutes to write but is incomprehensible six months later is not a productivity win. Verbose mode pays for itself on the first re-read.

The re.MULTILINE flag is another one that deserves special attention. Without it, ^ matches only at the very beginning of the entire string and $ matches only at the very end. With re.MULTILINE, they match at the beginning and end of each line within the string. This matters enormously when processing multi-line documents: searching for lines that start with a specific word, finding headings, or validating that each line in a file conforms to a format. If you find that ^ or $ anchors are not working as expected on multi-line text, the first thing to check is whether you need re.MULTILINE.

Let's build patterns for text elements you'll frequently encounter in NLP work. Real-world text contains entities such as emails and URLs, phone numbers and dates, plus social media elements. Understanding how to extract these is fundamental to text preprocessing.

Named Entity Recognition (NER) systems in production use a combination of machine learning models and rule-based regex patterns. The ML models handle ambiguous cases and understand context, while regex handles entities with well-defined formats. Phone numbers, email addresses, ZIP codes, ISBN numbers, credit card numbers, and ISO dates all have rigid formats that regex captures perfectly and ML models have no advantage on. The two approaches complement each other: regex for structure-defined entities, ML for semantically-defined entities.

Consider this sample social media post containing multiple entity types:

Let's examine each pattern in detail. Each pattern below is a working approximation, not a specification-perfect implementation. The formal standards for email addresses (RFC 5321) and URLs (RFC 3986) are far more complex than any practical regex. The patterns here cover the vast majority of real-world cases, which is exactly what you want for preprocessing: something that handles 99% of inputs correctly is far more useful than a formally complete pattern that is 200 characters long and impossible to maintain.

The email pattern works in three parts. The local part [A-Za-z0-9._%+-]+ allows letters, digits, and the special characters that email providers accept before the @. The domain part [A-Za-z0-9.-]+ covers subdomains and the domain name itself. The TLD \.[A-Z|a-z]{2,} requires at least two letters after the final dot, which excludes bare hostnames with no extension. Word boundaries at both ends prevent matching email addresses embedded inside longer tokens.

Notice that the URL pattern uses https? to accept both http and https, but not ftp or other schemes. The (?:www\.)? at the start makes the "www." prefix optional. The \b at the end of the domain portion prevents the pattern from matching mid-word, while the final non-capturing group (?:[-a-zA-Z0-9()@:%_\+.~#?&//=]*) captures the path, query string, and fragment. The * at the end is greedy and will consume the entire path, which is usually what you want when extracting full URLs.

The re.sub() function replaces matches with new text. You can use backreferences in the replacement string. Substitution is one of the most important regex operations for NLP preprocessing because cleaning text nearly always means replacing certain patterns with something else. You might replace URLs with a <URL> token so the model knows a URL was present without exposing the actual domain. You might replace all numbers with <NUM> to reduce vocabulary size. You might normalize whitespace, remove punctuation, or redact personally identifiable information (PII) before storing documents in a database.

The real power of re.sub() comes from its two modes: static replacement strings and callable replacement functions. A static string can reference captured groups using \1, \2, or named groups with \g<name>. A callable receives the match object and can execute arbitrary Python logic to compute the replacement, enabling transformations that would be impossible with a pure string replacement.

For patterns you use repeatedly, compile them for better performance. When you call re.findall(pattern, text) directly, Python must parse the pattern string and compile it into an internal representation every time that line runs. For a pattern called once in a script this cost is negligible, but in a loop that processes millions of documents, pattern compilation overhead adds up. re.compile() performs the compilation once and returns a pattern object that you call .findall(), .search(), .sub(), etc., on directly.

Compiled patterns also store flags, making your code cleaner when the same flags apply everywhere. A natural pattern is to compile all patterns at module load time as module-level constants. This ensures compilation happens exactly once for the lifetime of the process, and it makes the patterns easy to find and review.

Regex engines use backtracking to find matches, which can lead to catastrophic performance on certain patterns. To understand why, you need a basic picture of how a regex engine works. Most Python regex engines (including the one in the re module) use a backtracking NFA (nondeterministic finite automaton) approach. When the engine encounters a choice point, such as a quantifier that could match different numbers of characters, it picks one option, continues trying to match the rest of the pattern, and if that fails, backtracks to the choice point and tries another option. For well-designed patterns, this backtracking is limited and fast. For poorly designed patterns, the number of possible paths through the choice points grows exponentially with input length.

The logarithmic scale reveals the exponential nature of the problem. While the safe pattern stays flat (constant time), the dangerous pattern's execution time doubles with each additional character. At 20 characters, matching takes seconds. At 25, it takes minutes. At 30, hours. This is why avoiding nested quantifiers is critical for production code.

To write efficient regex patterns, follow these guidelines:

1. Anchor when possible: ^pattern is faster than searching the whole string
2. Be specific: [0-9] is faster than \d in some engines, [a-zA-Z] is faster than .
3. Avoid nested quantifiers: (a+)+ is dangerous; use a+ instead
4. Use non-capturing groups: (?:...) is slightly faster than (...)
5. Compile patterns: For repeated use, re.compile() avoids re-parsing
6. Use possessive quantifiers or atomic groups: Python's re doesn't support these, but the regex module does

Let's combine what we've learned into a practical text preprocessing pipeline. A cleaning pipeline is a sequence of regex operations applied in a specific order. Order matters: removing URLs before tokenizing prevents partial URL fragments from becoming spurious tokens. Normalizing whitespace after all substitutions ensures that multiple removals do not leave gaps with multiple spaces. The design principle is to always think about the pipeline as a sequence of transformations that progressively bring the text closer to the clean, normalized form your downstream model expects.

Regex is powerful, but it's not always the right tool. The key question to ask before reaching for regex is: "Does this problem involve a pattern that can be described as a sequence of character rules, or does it involve structure that can nest arbitrarily?" If the answer involves nesting (HTML tags inside other HTML tags, parentheses inside parentheses, functions inside function arguments), regex is the wrong tool. Regular expressions can only describe regular languages, which by definition cannot handle arbitrary nesting. Attempting to parse HTML with regex leads to one of the most famous pieces of Stack Overflow wisdom: do not do it.

Don't use regex for:

• HTML/XML parsing: Use BeautifulSoup or lxml. Regex can't handle nested structures properly. Even a "simple" task like extracting all link texts from an HTML document will break on edge cases: attributes with angle brackets, comments, CDATA sections, and nested elements all defeat naive regex approaches.
• JSON/structured data: Use the json module. JSON has recursive structure (objects inside arrays inside objects) and the standard library parses it correctly and safely. A regex that seems to work on your test data will fail on edge cases.
• Complex grammars: Use a proper parser (like pyparsing, lark, or antlr) for programming languages, configuration files, or any format with a formal grammar. Regex can tokenize, but it cannot parse.
• Simple string operations: str.split(), str.replace(), and the in operator are clearer and faster for simple cases. Reaching for regex when a string method suffices adds complexity and can mislead future readers about the intent.

The well-known saying in the NLP community is that regex is a "good servant but a bad master." Used for what it is designed for, it is elegant and powerful. Used as a universal text processing hammer, it creates unmaintainable, brittle code. Part of developing expertise with regex is developing the judgment to know when not to use it.

Regular expressions have fundamental limitations that you should understand before relying on them heavily. These are not just practical caveats. Some limitations are mathematically fundamental, rooted in the theory of formal languages. Understanding them will make you a more disciplined regex user and will help you recognize when you need a different tool entirely.

The most fundamental limitation is that regex can only describe regular languages: the class of languages recognized by finite automata. Many useful text patterns are not regular. Balanced parentheses are the classic example: no regex can accept exactly the strings with perfectly balanced () and reject all others. This is not a limitation of Python's regex engine. It is a mathematical impossibility. Any pattern that requires "counting" to an arbitrary depth, such as matching n opening delimiters followed by n closing ones, falls outside what regex can express. In practice, this means that regex is the wrong tool for anything with recursive structure: XML, JSON, programming language syntax, Markdown with nested lists.

The readability limitation is less fundamental but more practically damaging. Regex patterns are dense. Each character is meaningful, and there are almost no redundant characters to give the eye a place to rest. A pattern like (?<!\d)(\d{1,3}(?:,\d{3})*(?:\.\d+)?)\s*(?:USD|EUR|GBP) is a legitimate, useful pattern for extracting currency amounts. But without comments, it is essentially a write-only string. The re.VERBOSE flag with inline comments addresses this partially, but even with comments, long patterns remain difficult to audit for correctness. This creates a real maintenance risk: patterns that work are often left unchanged even as requirements evolve, because no one wants to touch them.

Unicode handling deserves special mention for NLP work. The \w shorthand in Python's re module matches Unicode letters and digits by default in Python 3, which is better than the ASCII-only behavior in Python 2. But it does not match all characters that might be considered "word characters" in every writing system. Emojis, for example, are not matched by \w. Some Unicode punctuation marks behave like word characters in certain scripts. If you are processing multilingual text, test your patterns carefully against text in all the languages you expect to encounter. The regex module provides access to Unicode character properties like \p{L} (any Unicode letter) and \p{N} (any Unicode digit), which are more reliable for multilingual processing than the standard \w shorthand.

The following limitations apply in practice:

• Context-free languages: Regex cannot match arbitrarily nested structures like balanced parentheses. No regex can match only strings with balanced parens like ((())).
• Readability: Complex regex patterns become write-only code. The email pattern we used earlier is already hard to read, and production-grade patterns are worse.
• Maintenance: Small changes to requirements can require complete pattern rewrites. Adding "support international characters" to an email pattern is non-trivial.
• Unicode complexity: While Python's re module handles Unicode, character classes like \w may not match all word characters in all languages. The regex module with Unicode categories helps.
• Performance unpredictability: Backtracking behavior makes it hard to predict execution time. A pattern that works fine on test data might hang on production data.

To bring together everything covered in this chapter, let's build a regex-based parser for server log files. Log parsing is one of the most common real-world regex tasks in data engineering and NLP pipelines: logs contain structured information embedded in text that varies in format across systems and versions. Regex is ideally suited for extracting the structured fields from each line.

A typical Apache web server log line looks like this:

This single line contains: an IP address, a username, a timestamp, an HTTP method and path, a status code, and a response size in bytes. Let's decompose the pattern needed to parse it.

The IP address consists of four groups of one to three digits separated by dots: (\d{1,3}\.){3}\d{1,3}. The username is a word following a dash and space: \S+ (any non-whitespace). The timestamp is enclosed in literal square brackets; the pattern uses \[ and \] to match those brackets, wrapping a group that captures the full timestamp string. The request line is in double quotes: the pattern captures the HTTP method and path separately from the HTTP version. The status code and response size are space-separated numbers at the end.

Combining these components into a named-group pattern:

Notice several things about this pattern. First, using re.VERBOSE makes the pattern readable despite its complexity. Each component has a comment explaining what it matches. Second, named groups mean that match.groupdict() returns a ready-to-use dictionary with meaningful keys. Third, the pattern fails gracefully on the invalid line: LOG_PATTERN.match() returns None, and the conditional check ensures we only process successful matches. This is the defensive style you should use in production: always check whether a match was found before accessing its groups.

The key insight from this example is that building complex patterns incrementally works better than trying to write the whole thing at once. Start with the IP address pattern and test it. Add the username component and test again. Continue adding components and testing after each addition. This incremental approach makes debugging far easier because each test tells you exactly which component of the pattern is failing.

When working with regular expressions in Python, these are the essential functions and their most important parameters:

re.search(pattern, string, flags=0)

• pattern: The regex pattern to search for
• string: The text to search within
• flags: Optional modifiers like re.IGNORECASE, re.MULTILINE
• Returns: A match object for the first match, or None if no match

re.findall(pattern, string, flags=0)

• Returns all non-overlapping matches as a list of strings
• If the pattern has groups, returns a list of tuples containing the groups

re.finditer(pattern, string, flags=0)

• Returns an iterator of match objects for all matches
• Use when you need position information or access to groups

re.sub(pattern, repl, string, count=0, flags=0)

• repl: Replacement string or function
• count: Maximum number of replacements (0 means all)
• Backreferences like \1 can be used in the replacement string

re.split(pattern, string, maxsplit=0, flags=0)

• maxsplit: Maximum number of splits (0 means no limit)
• Returns a list of strings split at pattern matches

re.compile(pattern, flags=0)

• Pre-compiles a pattern for repeated use
• Returns a compiled pattern object with the same methods

Common Flags

• re.IGNORECASE (or re.I): Case-insensitive matching
• re.MULTILINE (or re.M): ^ and $ match at line boundaries
• re.DOTALL (or re.S): . matches newlines
• re.VERBOSE (or re.X): Allow comments and whitespace in patterns

Regular expressions provide a compact, powerful language for pattern matching in text. They occupy a specific and important niche in the NLP practitioner's toolkit: they excel at extracting and transforming text with rigid structures, such as emails and phone numbers, dates and URLs, or log entries. They are not the right tool for problems involving nesting, context-dependence, or semantic understanding. Used appropriately, they are elegant and fast. Used inappropriately, they become unmaintainable liabilities.

The mental model that will serve you best is to think of regex patterns as specifications for text structure. You are describing the shape of what you are looking for, not writing a procedure to find it. The regex engine handles the search procedure. Your job is to describe the structure precisely enough that only the things you want match, and everything else does not. This requires careful attention to anchors (to prevent partial matches), character classes (to be specific about what is acceptable), quantifiers (to specify repetition correctly), and lookarounds (to use context as a filter without including it in results).

You've learned:

• Metacharacters: . matches any character, [] defines character classes, ^ and $ anchor to positions
• Quantifiers: *, +, ?, {n,m} control repetition; add ? for lazy matching
• Groups: () captures text, (?:) groups without capturing, (?P<name>) names captures
• Lookarounds: (?=), (?!), (?<=), (?<!) match positions based on context
• Backreferences: \1, \2 refer back to captured groups
• Flags: re.IGNORECASE, re.MULTILINE, re.DOTALL, re.VERBOSE modify behavior

Key practical patterns for NLP:

• Emails: \b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b
• URLs: https?://\S+
• Hashtags/Mentions: #\w+, @\w+
• Word boundaries: \bword\b for whole-word matching

Best practices:

• Always use raw strings: r"pattern"
• Compile patterns used repeatedly
• Avoid nested quantifiers that cause catastrophic backtracking
• Use string methods for simple operations
• Use proper parsers for structured formats like HTML or JSON
• Use re.VERBOSE with comments for any non-trivial pattern
• Use named groups in patterns you will maintain over time
• Test patterns on adversarial inputs and the happy path

In the next chapter, we'll explore sentence segmentation, where regex plays a supporting role in identifying sentence boundaries. You'll see how the word boundary anchor \b and end-of-sentence punctuation patterns combine with heuristics to split continuous text into discrete sentences, a foundational step in almost every NLP pipeline.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about regular expressions in Python.