Web Crawling › Summary

Web crawling provides the raw material for the largest language model training corpora. The core loop is simple: fetch URLs, extract links, repeat. The complexity comes from doing this at scale while respecting servers, managing quality, and complying with legal constraints.

Key takeaways from this chapter:

• Common Crawl is the foundational public data source for language AI, giving petabyte-scale WARC, WAT, and WET data with monthly crawl releases. WET files provide pre-extracted plain text for NLP use cases, while WARC files preserve the complete raw crawl data for custom text extraction.
• Robots.txt defines the social contract between sites and crawlers. Respecting it is both an ethical obligation and practically essential for sustainable crawling. The protocol includes allow/disallow rules, crawl delay directives, and sitemap pointers.
• Politeness policies prevent a single crawler from overwhelming servers. Per-host rate limiting, crawl delay respect, adaptive backoff on server errors, and responsible user-agent identification are minimum requirements.
• Crawl strategy shapes data character. Breadth-first crawls produce broad coverage; focused crawls produce topically relevant but narrower datasets. Depth and freshness tradeoffs further determine what ends up in the corpus.
• Coverage bias is inherent: web crawls miss paywalled, dynamic, and login-required content, skewing the resulting data toward publicly accessible, often English-language material. JavaScript rendering, domain imbalance, and temporal non-uniformity create additional biases that practitioners must understand and manage.
• Data quality varies enormously across the web. Content length filtering is a fast first pass, but the full pipeline requires deduplication, language identification, quality filtering, and toxicity filtering, topics we turn to next.

The next chapter on Document Extraction explores how to reliably convert raw HTML from crawl data into clean text, a step that proves more difficult than it first appears given the diversity of web page structures and encodings.