Web Crawling › How Web Crawlers Work

Web crawlers operate a simple loop: take a URL, download the page at that URL, extract all links from the page, add new links to a queue, and repeat. The complexity comes from doing this at scale, doing it without overwhelming servers, and doing it in a way that produces high-quality, well-organized data.

The conceptual simplicity of this loop is deceptive. A production crawler must handle hundreds of millions of URLs daily, deal with malformed HTML, follow redirects, manage connection pooling, cache DNS results, respect per-host rate limits, parse and obey robots.txt rules, detect and skip duplicate pages, and write output in formats that downstream pipelines can process efficiently. Every one of these concerns interacts with the others. A DNS cache that is too aggressive causes stale IP addresses that lead to connection failures. A rate limiter that is too loose causes site operators to block the crawler's IP. An HTML parser that fails silently on malformed markup drops entire pages from the corpus without warning.

This section walks through the core components of the crawl process: seed selection, the main loop, URL normalization, DNS and HTTP mechanics, and frontier management at scale.

Before any crawling begins, a crawler needs a starting point. Seed URLs are the initial set of pages from which the crawler begins following links. The choice of seeds has an outsized influence on what the final crawl contains, because breadth-first traversal explores the neighborhood of seeds first and most thoroughly.

For general-purpose crawls like Common Crawl, seeds typically come from several sources. Authoritative web directories, such as the DMOZ Open Directory (now archived) or curated lists of major news sites, government portals, and educational institutions, provide high-quality starting points. URLs from previous crawls that were productive are re-seeded to maintain continuity. XML sitemaps published by individual sites enumerate pages that owners want indexed, giving a direct invitation to crawl. Domain registrar data reveals newly registered domains that may contain fresh content.

The seed strategy creates a quality bias from the start. A crawl seeded from Wikipedia will produce data richer in factual reference content than a crawl seeded from social media domains. A crawl seeded from curated academic directories will skew toward formal writing and technical vocabulary. This quality inheritance, where crawled content mirrors the editorial standards of the seeded domains, is a feature that language AI practitioners exploit deliberately when building specialized corpora.

The number of seeds also matters. A few hundred seed URLs in a breadth-first crawler will naturally cluster the crawl around those seeds' neighborhoods. A million seeds, carefully selected to span diverse languages, topics, and regions, produces much more balanced coverage. Common Crawl uses tens of millions of seeds across a wide variety of domains and languages, which is part of why its coverage is unusually broad compared to purpose-built enterprise crawlers.

For a focused crawl building a domain-specific corpus, seeds are chosen differently. If you want a dataset of Python programming content, you might seed from the official Python documentation, PyPI project pages, Stack Overflow Python questions, major Python blogs, and key GitHub repositories. These seeds will guide the link-following traversal toward the neighborhood of Python content. You might also feed the crawler a keyword list and have it score discovered URLs by the presence of those keywords, making sure that pages added to the frontier are relevant to your target domain.

The fundamental crawl algorithm proceeds through a series of well-defined steps. The crawler begins with a set of seed URLs, typically highly authoritative pages like major news sites, Wikipedia, or web directories. For each URL in the queue, the crawler issues an HTTP GET request and stores the response. It then parses the response to extract hyperlinks using the href attributes in anchor tags. Each new URL that has not already been visited and passes acceptability filters gets added to the frontier, the data structure holding URLs yet to visit. This loop continues until either the frontier is exhausted or some stopping condition, like a document count or time limit, is reached.

The simplest frontier is a FIFO queue, which produces a breadth-first traversal. Start from seeds, visit all pages linked from seeds, then all pages linked from those pages, and so on. Breadth-first crawling naturally covers the most highly connected pages first, since popular pages appear as targets in many early crawls.

Depth-first traversal, using a stack instead of a queue, follows link chains deep into individual sites before moving on. This strategy is more useful for focused crawls that want to exhaust a specific domain's content, but it risks getting stuck in link traps: cycles of dynamically generated pages that extend indefinitely. A product catalog with 10,000 items and a sorting parameter, a calendar system generating every possible date URL, or a search results page with query string variations can all trap a depth-first crawler in a technically infinite loop if the crawler does not implement loop detection.

Production crawlers typically use priority queues rather than simple FIFO structures. URLs are scored based on predicted importance, recency, or content type, and higher-scored URLs are downloaded first. This allows the crawler to be more selective about what it visits when resources are limited. A priority crawler might assign higher scores to URLs from domains that have historically produced high-quality content, URLs that contain specific keywords relevant to the crawl's focus, or URLs that appear as link targets on many different pages, a signal of importance analogous to PageRank.

The web is full of URLs that point to the same content. Consider these URLs:

• http://example.com/page
• HTTP://EXAMPLE.COM/PAGE
• http://example.com/page?utm_source=twitter
• http://example.com/page#section2
• http://www.example.com/page

All of these may return identical or nearly identical content. Without normalization, a crawler visits the same page multiple times under different URLs, wasting bandwidth and cluttering the dataset with duplicates. URL normalization applies a sequence of transformations to produce a canonical form: lowercasing the scheme and host, sorting query parameters, removing fragment identifiers, stripping known tracking parameters like utm_source and fbclid, and resolving relative URLs against the page's base URL.

URL normalization is more subtle than it appears. Some query parameters are semantically significant, like ?lang=en or ?page=2, while others are pure tracking noise. Stripping the wrong parameters produces false merges where two different pages get collapsed to the same canonical URL. Keeping too many tracking parameters produces false splits where many crawl attempts hit the same actual page. Production crawlers maintain lists of known tracking parameters to strip and use heuristics for unknown parameters based on their structure and context.

Even after normalization, URL deduplication alone is insufficient. Many websites serve the same content at multiple canonical URLs through redirects, mirrors, or content management system artifacts. A news article might be accessible at /2024/03/article-title, /article/12345, and /amp/article-12345, all returning essentially the same text. For this reason, content-level deduplication, which we will explore in the Deduplication chapter, is equally important downstream.

Link traps deserve special attention. Some websites generate infinite URLs intentionally or accidentally. A calendar application might produce URLs for every possible date. A dynamic filter interface might allow arbitrary combinations of filtering parameters, each yielding a unique URL. E-commerce sites with faceted search can generate billions of unique-looking URLs from a few dozen actual products. Crawlers defend against link traps with several heuristics: URL length limits, maximum path depth, pattern detection for parametric URL explosions, and maximum pages-per-domain caps.

For each URL, the crawler must resolve the hostname to an IP address using the Domain Name System, then open a TCP connection and issue an HTTP request. This two-step process sounds straightforward but becomes a significant bottleneck at scale.

DNS resolution is inherently network-dependent. Resolving a single hostname may take anywhere from 1 millisecond for a cached entry to several hundred milliseconds for a cold lookup against distant authoritative name servers. At web scale, with millions of unique hostnames being crawled, DNS resolution latency accumulates to a substantial fraction of total crawl time. Production crawlers cache DNS results aggressively, typically with TTLs of several hours, and batch DNS lookups for new domains using asynchronous DNS libraries that can resolve thousands of hostnames concurrently. Some large-scale crawlers maintain their own authoritative DNS resolvers to reduce latency further.

HTTP connections carry significant overhead from TCP handshake and TLS negotiation. A full TLS 1.3 handshake requires two round trips before any application data can be sent. Crawlers reuse connections via HTTP keep-alive within the same host and batch requests to the same IP to amortize connection costs. HTTP/2 support provides multiplexed connections where multiple requests share a single TCP connection, further reducing per-request overhead. At the scale of millions of pages per day, even small per-request latency improvements translate to meaningful increases in crawl throughput.

Response handling requires checking status codes, and each category demands a different response. A 200 OK means success; the content can be stored. A 301 Moved Permanently or 302 Found means the page has moved; the crawler follows the redirect URL and records the mapping between old and new URL. A 404 Not Found means the page no longer exists; the crawler records this and removes the URL from future crawl schedules. A 429 Too Many Requests means the server is throttling requests; the crawler must back off exponentially and try again later. A 503 Service Unavailable means temporary unavailability; the crawler queues the URL for a retry after a delay. A 5xx error from a server generally indicates a transient problem, and well-designed crawlers implement exponential backoff with jitter for such failures to avoid thundering-herd effects when many crawl workers hit the same server simultaneously.

Content-type negotiation matters too. Crawlers should request text/html and handle other content types appropriately. A URL that returns a PDF needs different processing than one that returns HTML. Some crawlers include PDF and EPUB extraction pipelines for high-quality documents, while others skip non-HTML content types entirely for simplicity.

Robots.txt files tell crawlers to avoid certain paths entirely, and well-behaved crawlers check and respect these instructions before fetching any URL on a domain. We discuss robots.txt in detail in the next section.

For small crawls, an in-memory queue is sufficient. At web scale, the frontier can contain billions of URLs that must be managed persistently across machine restarts. Production crawlers like those operated by Google or Common Crawl use distributed frontier managers backed by databases or specialized queue systems.

A key design challenge is balancing crawl politeness against crawl speed. You cannot send a thousand requests per second to a single server. Instead, the frontier must ensure that no server receives requests faster than its politeness policy allows. This typically means grouping URLs by host, and for each host, maintaining a separate subqueue with delays between requests. The classic approach partitions the frontier into per-host buckets and draws one URL from each bucket in round-robin fashion, inserting a mandatory wait before re-visiting any bucket.

Distributed crawlers add further complexity. When many worker machines are fetching pages in parallel, the shared frontier must coordinate so that no two workers attempt to fetch the same URL simultaneously. This requires either a centralized frontier service with high availability, or a partitioned frontier where URL assignments are sharded by host and each shard is owned by exactly one worker. Common Crawl's crawler infrastructure, for example, uses a distributed queue system that ensures consistent URL assignment across hundreds of parallel workers while maintaining per-host politeness.

The frontier also maintains metadata for each URL beyond the URL string itself. Crawlers track the URL's discovery timestamp, its link depth from the nearest seed, its estimated relevance score, its crawl priority, and any known metadata from sitemaps. This metadata drives scheduling decisions that determine which pages get crawled and when.