Web Crawling › Web Crawling

The internet is the largest corpus of human knowledge ever assembled. Hundreds of billions of web pages document everything from scientific research to casual conversation, from formal legislation to informal social media posts. When researchers at Common Crawl or AI labs want to train a language model on this knowledge, they face a fundamental engineering challenge: how do you systematically discover, download, and store that content at a scale that spans petabytes of data?

Web crawling is the answer. A web crawler, also called a spider or bot, is a program that automatically traverses the web by following hyperlinks, downloading page content as it goes. The same technology that powers Google's search index also underpins the training data pipelines for virtually every large language model trained today. GPT-3 used filtered Common Crawl data for roughly 60% of its training corpus. LLaMA, Falcon, and Mistral all draw heavily from crawled web text. Understanding how web crawling works, where its data comes from, and how to respect its social contract is foundational knowledge for anyone building modern language AI systems.

This chapter covers the mechanics of web crawling, from seed URL discovery through frontier management and politeness policies. We examine Common Crawl in depth because it is the single most important public data source in language AI. We then explore how freshness, depth, and breadth tradeoffs shape the character of crawled datasets, and we work through a hands-on implementation of a focused crawler. By the end, you will understand how crawling works mechanically and why every design decision, from seed selection to rate limiting, has downstream consequences for the quality and character of the training data you produce.