Regular Expressions › Worked Example: Building a Log Parser
Language AI HandbookAll chapters Regular Expressions › Worked Example: Building a Log ParserOne section · Sections in this chapter18 / 21 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
To bring together everything covered in this chapter, let's build a regex-based parser for server log files. Log parsing is one of the most common real-world regex tasks in data engineering and NLP pipelines: logs contain structured information embedded in text that varies in format across systems and versions. Regex is ideally suited for extracting the structured fields from each line.
A typical Apache web server log line looks like this:
203.0.113.42 - alice [15/Jan/2024:10:22:45 -0500] "GET /api/data HTTP/1.1" 200 1234
This single line contains: an IP address, a username, a timestamp, an HTTP method and path, a status code, and a response size in bytes. Let's decompose the pattern needed to parse it.
The IP address consists of four groups of one to three digits separated by dots: (\d{1,3}\.){3}\d{1,3}. The username is a word following a dash and space: \S+ (any non-whitespace). The timestamp is enclosed in literal square brackets; the pattern uses \[ and \] to match those brackets, wrapping a group that captures the full timestamp string. The request line is in double quotes: the pattern captures the HTTP method and path separately from the HTTP version. The status code and response size are space-separated numbers at the end.
Combining these components into a named-group pattern:
import re
LOG_PATTERN = re.compile(
r"""
(?P<ip>\d{1,3}(?:\.\d{1,3}){3}) # IP address
\s+-\s+ # Separator dash
(?P<user>\S+) # Username
\s+
\[(?P<timestamp>[^\]]+)\] # Timestamp in brackets
\s+
"(?P<method>[A-Z]+) # HTTP method
\s+(?P<path>\S+) # Request path
\s+HTTP/[\d.]+" # HTTP version
\s+
(?P<status>\d{3}) # Status code
\s+
(?P<size>\d+) # Response size in bytes
""",
re.VERBOSE,
)
sample_logs = [
'203.0.113.42 - alice [15/Jan/2024:10:22:45 -0500] "GET /api/data HTTP/1.1" 200 1234',
'198.51.100.7 - bob [15/Jan/2024:10:22:50 -0500] "POST /login HTTP/1.1" 302 0',
'192.0.2.89 - - [15/Jan/2024:10:23:01 -0500] "GET /static/app.js HTTP/1.1" 304 0',
"invalid log line that will not match",
]
parsed_logs = []
for line in sample_logs:
match = LOG_PATTERN.match(line)
if match:
parsed_logs.append(match.groupdict())Parsed 3 of 4 log lines successfully. IP: 203.0.113.42 User: alice Timestamp: 15/Jan/2024:10:22:45 -0500 Method: GET Path: /api/data Status: 200 Size: 1234 bytes IP: 198.51.100.7 User: bob Timestamp: 15/Jan/2024:10:22:50 -0500 Method: POST Path: /login Status: 302 Size: 0 bytes IP: 192.0.2.89 User: - Timestamp: 15/Jan/2024:10:23:01 -0500 Method: GET Path: /static/app.js Status: 304 Size: 0 bytes
Notice several things about this pattern. First, using re.VERBOSE makes the pattern readable despite its complexity. Each component has a comment explaining what it matches. Second, named groups mean that match.groupdict() returns a ready-to-use dictionary with meaningful keys. Third, the pattern fails gracefully on the invalid line: LOG_PATTERN.match() returns None, and the conditional check ensures we only process successful matches. This is the defensive style you should use in production: always check whether a match was found before accessing its groups.
The key insight from this example is that building complex patterns incrementally works better than trying to write the whole thing at once. Start with the IP address pattern and test it. Add the username component and test again. Continue adding components and testing after each addition. This incremental approach makes debugging far easier because each test tells you exactly which component of the pattern is failing.