Regular Expressions › Common NLP Patterns
Language AI HandbookAll chapters Regular Expressions › Common NLP PatternsOne section · Sections in this chapter11 / 21 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Let's build patterns for text elements you'll frequently encounter in NLP work. Real-world text contains entities such as emails and URLs, phone numbers and dates, plus social media elements. Understanding how to extract these is fundamental to text preprocessing.
Named Entity Recognition (NER) systems in production use a combination of machine learning models and rule-based regex patterns. The ML models handle ambiguous cases and understand context, while regex handles entities with well-defined formats. Phone numbers, email addresses, ZIP codes, ISBN numbers, credit card numbers, and ISO dates all have rigid formats that regex captures perfectly and ML models have no advantage on. The two approaches complement each other: regex for structure-defined entities, ML for semantically-defined entities.
Consider this sample social media post containing multiple entity types:
import re
sample_text = """
Hey @john_doe! Check out our new product at https://example.com/product?id=123
Contact us at support@company.com or call (555) 123-4567 for help.
Sale ends 2024-12-31! Use code #SAVE20 for 20% off.
Also follow @tech_news and @deals_daily for updates.
Visit http://blog.example.org or email sales@example.org
#BlackFriday #CyberMonday #Shopping
Meeting scheduled for 01/15/2024. Call +1-800-555-0199.
"""
# Define extraction patterns
patterns = {
"Emails": r"\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b",
"URLs": r"https?://\S+",
"Mentions": r"@\w+",
"Hashtags": r"#\w+",
"Phone Numbers": r"(?:\+?1[-.\s]?)?\(?\d{3}\)?[-.\s]?\d{3}[-.\s]?\d{4}",
"Dates (ISO)": r"\d{4}-\d{2}-\d{2}",
"Dates (US)": r"\d{2}/\d{2}/\d{4}",
}
# Extract all entities
for name, pattern in patterns.items():
matches = re.findall(pattern, sample_text)Entity extraction results:
Emails: 2 matches
Examples: 'support@company.com', 'sales@example.org'
URLs: 2 matches
Examples: 'https://example.com/product?id=123', 'http://blog.example.org'
Mentions: 5 matches
Examples: '@john_doe', '@company', ...
Hashtags: 4 matches
Examples: '#SAVE20', '#BlackFriday', ...
Phone Numbers: 2 matches
Examples: '(555) 123-4567', '+1-800-555-0199'
Dates (ISO): 1 matches
Examples: '2024-12-31'
Dates (US): 1 matches
Examples: '01/15/2024'The table below summarizes the extraction results. Notice how social media content tends to have many mentions and hashtags, while business communications include emails and phone numbers:
| Entity Type | Count | Example Matches |
|---|---|---|
| Emails | 2 | support@company.com, sales@example.org |
| URLs | 2 | https://example.com/product?id=123, http://blog.example.org |
| Mentions | 3 | @john_doe, @tech_news, @deals_daily |
| Hashtags | 4 | #SAVE20, #BlackFriday, #CyberMonday, #Shopping |
| Phone Numbers | 2 | (555) 123-4567, +1-800-555-0199 |
| Dates (ISO) | 1 | 2024-12-31 |
| Dates (US) | 1 | 01/15/2024 |
Let's examine each pattern in detail. Each pattern below is a working approximation, not a specification-perfect implementation. The formal standards for email addresses (RFC 5321) and URLs (RFC 3986) are far more complex than any practical regex. The patterns here cover the vast majority of real-world cases, which is exactly what you want for preprocessing: something that handles 99% of inputs correctly is far more useful than a formally complete pattern that is 200 characters long and impossible to maintain.
Email Addresses
# Basic email pattern
email_pattern = r"\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b"
text = """
Contact us at support@example.com or sales@company.org.
Invalid emails: @missing.com, nodomain@, spaces in@email.com
Edge cases: user.name+tag@sub.domain.co.uk
"""
emails = re.findall(email_pattern, text)Email pattern:
\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b
Found emails:
support@example.com
sales@company.org
in@email.com
user.name+tag@sub.domain.co.ukThe email pattern works in three parts. The local part [A-Za-z0-9._%+-]+ allows letters, digits, and the special characters that email providers accept before the @. The domain part [A-Za-z0-9.-]+ covers subdomains and the domain name itself. The TLD \.[A-Z|a-z]{2,} requires at least two letters after the final dot, which excludes bare hostnames with no extension. Word boundaries at both ends prevent matching email addresses embedded inside longer tokens.
URLs
# URL pattern
url_pattern = r"https?://(?:www\.)?[-a-zA-Z0-9@:%._\+~#=]{1,256}\.[a-zA-Z0-9()]{1,6}\b(?:[-a-zA-Z0-9()@:%_\+.~#?&//=]*)"
text = """
Visit https://www.example.com or http://test.org/page?id=123
Also check https://api.service.io/v2/users#section
Not a URL: ftp://other.com or just example.com
"""
urls = re.findall(url_pattern, text)URL pattern (simplified): https?://... Found URLs: https://www.example.com http://test.org/page?id=123 https://api.service.io/v2/users#section
Notice that the URL pattern uses https? to accept both http and https, but not ftp or other schemes. The (?:www\.)? at the start makes the "www." prefix optional. The \b at the end of the domain portion prevents the pattern from matching mid-word, while the final non-capturing group (?:[-a-zA-Z0-9()@:%_\+.~#?&//=]*) captures the path, query string, and fragment. The * at the end is greedy and will consume the entire path, which is usually what you want when extracting full URLs.
Phone Numbers
# US phone number patterns (multiple formats)
phone_pattern = r"""
(?:
\+?1[-.\s]? # Optional country code
)?
(?:
\(?\d{3}\)? # Area code with optional parens
[-.\s]? # Separator
)
\d{3} # First 3 digits
[-.\s]? # Separator
\d{4} # Last 4 digits
"""
text = """
Call us: (555) 123-4567, 555.123.4567, 555 123 4567
International: +1-555-123-4567, +1 (555) 123-4567
"""
phones = re.findall(phone_pattern, text, re.VERBOSE)Phone numbers found: '(555) 123-4567' '555.123.4567' '555 123 4567' '+1-555-123-4567' '+1 (555) 123-4567'
Dates
text = """
Dates: 2024-01-15, 01/15/2024, January 15, 2024
Also: 15-Jan-2024, 15 January 2024
"""
# ISO format: YYYY-MM-DD
iso_dates = re.findall(r"\d{4}-\d{2}-\d{2}", text)
# US format: MM/DD/YYYY
us_dates = re.findall(r"\d{2}/\d{2}/\d{4}", text)
# Month name formats
month_names = r"(?:January|February|March|April|May|June|July|August|September|October|November|December|Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)"
named_dates = re.findall(
rf"(?:\d{{1,2}}[-\s])?{month_names}[-\s]?\d{{1,2}}(?:,?\s*\d{{4}})?", text
)Text: Dates: 2024-01-15, 01/15/2024, January 15, 2024 Also: 15-Jan-2024, 15 January 2024 ISO dates (YYYY-MM-DD): ['2024-01-15'] US dates (MM/DD/YYYY): ['01/15/2024'] Named dates: ['January 15, 2024', '15-Jan-20', '15 January 20']
Hashtags and Mentions
tweet = "Just learned about #NLP and #MachineLearning! Thanks @professor_ai for the great tutorial. #AI2024"
# Hashtags: # followed by word characters
hashtags = re.findall(r"#\w+", tweet)
# Mentions: @ followed by word characters
mentions = re.findall(r"@\w+", tweet)Tweet: 'Just learned about #NLP and #MachineLearning! Thanks @professor_ai for the great tutorial. #AI2024' Hashtags: ['#NLP', '#MachineLearning', '#AI2024'] Mentions: ['@professor_ai']