Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Web Crawling › HTTP Status Code Patterns in Crawls

Michael BrenndoerferJanuary 8, 20263 min read

Language AI HandbookAll chapters

Part 20 of 67Data Curation

Chapter 139Web Crawling › HTTP Status Code Patterns in Crawls

Reading section 10 of 13 · HTTP Status Code Patterns in Crawls · 3 minSection 10 of 13 · 3 min

Section 10 of 13 · 3 min read

Web Crawling › HTTP Status Code Patterns in CrawlsOne section · Sections in this chapter10 / 13 ↓

Sections in this chapter13

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Understanding how status codes distribute across a real crawl reveals important characteristics of the web that affect training data quality. Let us visualize what a typical status code distribution looks like:

In[16]:

Code

# Build a simulated status code distribution based on empirical crawl data
# Distributions reflect typical web crawl observations from Common Crawl reports
status_categories = {
    "200 OK": 0.72,
    "301/302 Redirect": 0.11,
    "404 Not Found": 0.09,
    "403 Forbidden": 0.04,
    "429 Rate Limited": 0.02,
    "5xx Server Error": 0.015,
    "Other": 0.005,
}

# Simulate counts for a crawl of 10000 URLs
total_attempted = 10_000
status_counts = {
    k: int(v * total_attempted) for k, v in status_categories.items()
}

# Compute pages actually usable for training
usable = status_counts["200 OK"]
usable_pct = usable / total_attempted * 100

Out[17]:

Visualization

Horizontal bar chart showing HTTP status code categories by percentage of total crawl attempts. — Simulated HTTP status code distribution for a web crawl of 10,000 URLs, based on patterns observed in large-scale crawls. Successful responses (200 OK) account for roughly 72% of attempts. Redirects account for an additional 11%, many of which ultimately resolve to successful pages. Together, the non-2xx responses reveal that nearly 30% of crawl bandwidth is spent on pages that cannot directly contribute usable training data, reinforcing the importance of efficient URL prioritization.

The status code breakdown exposes the real efficiency of a web crawl. Only pages with 200 OK status codes directly contribute training data, and even among those, quality filtering will remove a significant fraction. The 11% redirect rate represents URLs where the content has moved; a crawler that follows redirects correctly can recover some of this content, but chains of more than two or three redirects often lead to low-quality destinations or infinite redirect loops. The 9% 404 rate represents dead links in the web graph, which is unavoidable: the web changes constantly and old links break.