Web Crawling › Crawl Strategies

Not all crawls are created equal. The strategy a crawler uses shapes the character of the resulting dataset in fundamental ways. The tradeoffs between breadth, depth, freshness, and topical focus determine what kinds of knowledge end up in the data.

A breadth-first crawler makes no assumptions about what topics are valuable. It follows all links equally, expanding outward from seeds in all directions. This produces broad coverage of the web graph but shallow coverage of any individual site or topic. Common Crawl uses a broad, general-purpose strategy: it wants to cover as much of the web as possible across all languages and topics. The resulting dataset reflects the web's actual distribution of content, which means a lot of English-language commercial and social media content.

A focused crawler, by contrast, prioritizes pages that match a specific topic or domain. It uses classifiers or keyword heuristics to score each URL before adding it to the frontier, then visits high-scoring URLs first. A focused crawl on scientific literature would prioritize domains like arxiv.org, pubmed.ncbi.nlm.nih.gov, and academic publishers. A focused crawl on programming would prioritize Stack Overflow, GitHub, and documentation sites. A focused crawl on legal text would target court opinion databases, regulatory archives, and law review journals.

The advantage of focused crawling is efficiency: you download far fewer irrelevant pages to assemble a domain-specific corpus. If you want a high-quality programming dataset, starting from Common Crawl and filtering is one approach, but directly crawling programming-relevant domains is often more efficient and produces cleaner results. The disadvantage of focused crawling is that topic classifiers can miss relevant content that uses unexpected terminology or lives on unexpected domains. A blog post about machine learning hosted on a personal site that the crawler has never seen might score poorly on a classifier trained on known ML domains.

Hybrid strategies combine the two approaches. Some crawlers use broad crawling to discover new domains and focused policies to determine crawl priority within each domain. Common Crawl itself applies some form of quality and relevance scoring to prioritize crawl bandwidth toward higher-quality pages.

Link distance from seed pages is a strong proxy for content quality. Pages that are one click from a trusted seed, like Wikipedia or a major news site, tend to be more relevant and authoritative than pages buried six hops deep in the web graph. Common Crawl seeds from web directories and past crawl data, which means its coverage is richer for well-connected sites and sparser for isolated niche domains.

Crawl depth, the maximum number of links followed from a seed URL, directly controls how far into the long tail the crawler reaches. A shallow crawl at depth 2 or 3 captures mainstream content efficiently. A deep crawl can surface rare or specialized content that shallow crawls miss, but at the cost of downloading much more spam and low-quality content.

The relationship between crawl depth and content quality reflects how the web's link structure works. High-quality, authoritative content tends to receive many inbound links from other authoritative pages. This is exactly what PageRank, the algorithm underlying Google's original search ranking, measures: the probability that a random web surfer lands on a page by following links. Pages with high PageRank appear early in breadth-first traversals. Training data composed primarily of high-PageRank pages tends to be more coherent and authoritative than data from the long tail.

The practical implication is that very deep crawls, while theoretically yielding more content, often produce diminishing returns for training data quality. Each additional hop into the web graph brings in more spam, low-quality content, and link-farm pages that are only there to manipulate search rankings. Most high-quality training data pipelines cap crawl depth and apply quality filters to compensate, rather than trying to recover quality from a very deep crawl.

The web changes constantly. Pages are updated, deleted, and created at rates that vary enormously by site type. News sites publish dozens of articles per day. Academic papers are rarely updated after publication. Social media posts are ephemeral. A static snapshot of the web taken today is already partially stale tomorrow.

Freshness policies govern how often a crawler revisits pages it has previously seen. There are several common approaches. The simplest is a fixed-interval policy: revisit every page every $N$ days regardless of content. A more sophisticated approach is change-rate estimation: track how often a page's content changes and revisit it proportionally more often if it changes frequently. A site that publishes multiple articles per day warrants daily revisits, while a site whose content changes once a year warrants annual revisits.

For language model training data, freshness has subtle implications. Models trained on more recent data know about events and terminology that did not exist when older data was collected. However, very recent data may include lower-quality content, as the web accumulates more ephemeral, low-effort content over time. Common Crawl's monthly release cadence provides a reasonable freshness tradeoff for training data pipelines that can select crawls by date.

The freshness-versus-bandwidth tradeoff is unavoidable. Revisiting all pages frequently requires enormous crawler bandwidth. In practice, production crawlers use adaptive scheduling that prioritizes fresh revisits to high-value, frequently-changing pages and less frequent revisits to stable content. The optimal revisit interval for a page is proportional to the page's change rate. Pages that change daily should be revisited daily; pages that change annually need only annual revisits. Some crawlers model this explicitly using exponential smoothing of observed change rates, while others use simpler heuristics based on content type.