Regular Expressions › When Not to Use Regex

Michael BrenndoerferUpdated March 18, 20253 min read
Language AI HandbookAll chapters
Part 1 of 67Text as Data
Chapter 3Regular Expressions › When Not to Use Regex
Reading section 16 of 21 · When Not to Use Regex · 3 minSection 16 of 21 · 3 min

Book contents

Language AI Handbook67 parts · 513 chaptersView the whole book →
Active chapter 3 of 513

Section 16 of 21 · 3 min read

Regular Expressions › When Not to Use RegexOne section · Sections in this chapter16 / 21 ↓
Sections in this chapter21
Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Regex is powerful, but it's not always the right tool. The key question to ask before reaching for regex is: "Does this problem involve a pattern that can be described as a sequence of character rules, or does it involve structure that can nest arbitrarily?" If the answer involves nesting (HTML tags inside other HTML tags, parentheses inside parentheses, functions inside function arguments), regex is the wrong tool. Regular expressions can only describe regular languages, which by definition cannot handle arbitrary nesting. Attempting to parse HTML with regex leads to one of the most famous pieces of Stack Overflow wisdom: do not do it.

Don't use regex for:

  • HTML/XML parsing: Use BeautifulSoup or lxml. Regex can't handle nested structures properly. Even a "simple" task like extracting all link texts from an HTML document will break on edge cases: attributes with angle brackets, comments, CDATA sections, and nested elements all defeat naive regex approaches.
  • JSON/structured data: Use the json module. JSON has recursive structure (objects inside arrays inside objects) and the standard library parses it correctly and safely. A regex that seems to work on your test data will fail on edge cases.
  • Complex grammars: Use a proper parser (like pyparsing, lark, or antlr) for programming languages, configuration files, or any format with a formal grammar. Regex can tokenize, but it cannot parse.
  • Simple string operations: str.split(), str.replace(), and the in operator are clearer and faster for simple cases. Reaching for regex when a string method suffices adds complexity and can mislead future readers about the intent.

The well-known saying in the NLP community is that regex is a "good servant but a bad master." Used for what it is designed for, it is elegant and powerful. Used as a universal text processing hammer, it creates unmaintainable, brittle code. Part of developing expertise with regex is developing the judgment to know when not to use it.

In[69]:
Code
# Simple cases: prefer string methods
text = "Hello, World!"

# Bad: regex for simple check
uses_regex = bool(re.search(r"World", text))

# Good: simple 'in' operator
uses_in = "World" in text

# Bad: regex for simple replacement
regex_replace = re.sub(r",", ";", text)

# Good: str.replace()
str_replace = text.replace(",", ";")

# Bad: regex for simple split
regex_split = re.split(r",\s*", "a, b, c")

# Good: str.split() with strip
str_split = [x.strip() for x in "a, b, c".split(",")]
Out[70]:
Console
Prefer string methods for simple operations:

Checking substring:
  'in' operator: True (clearer)

Simple replacement:
  str.replace(): 'Hello; World!' (faster)

Simple split:
  str.split(): ['a', 'b', 'c'] (more readable)