Web Crawling › Common Crawl
Language AI HandbookAll chapters Web Crawling › Common CrawlOne section · Sections in this chapter4 / 13 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Common Crawl is a nonprofit organization that has been crawling the web since 2007 and making its data freely available. It has become the single most important source of training data for large language models. Understanding Common Crawl's structure, scale, and quality characteristics is essential for anyone building language AI systems.
Scale and Coverage
Common Crawl releases new crawls roughly monthly, each containing 3 to 5 billion web pages. The archive now holds over 250 billion pages accumulated over more than 15 years. The raw data is stored in WARC (Web ARChive) format, a standard format for web archives that preserves both HTTP response headers and page content. Each monthly crawl is approximately 70 to 100 terabytes of WARC files, hosted on Amazon S3 where they are accessible at no cost for data transfer within AWS and at very low cost for external downloads.
The coverage of Common Crawl is deliberately broad rather than deep. It does not attempt to crawl every page on every site exhaustively. Instead, it prioritizes popular, linked-to pages across a wide variety of domains and languages. The result is good coverage of the most important content on the internet, with sparser coverage of niche or low-traffic sites.
This breadth-over-depth strategy has clear implications for language AI. Common Crawl data captures mainstream web writing well: news articles, Wikipedia, Stack Overflow, Reddit, GitHub, and major blogs are all well represented. Niche academic forums, regional news in underrepresented languages, and community wikis may appear rarely or not at all. Understanding these coverage gaps is important when evaluating what knowledge a model trained on Common Crawl data will and will not possess.
WARC (Web ARChive) is a file format for storing web crawl data. Each WARC record contains the full HTTP response for a single URL, including headers and body content, along with metadata like the crawl timestamp and target URL. WARC is an ISO standard (ISO 28500) commonly used by web archives and Common Crawl.
Common Crawl Data Products
Common Crawl publishes three main data products, each serving a different use case in the data processing pipeline.
WARC files are the primary artifact from which all other products are derived. Each WARC file is a compressed sequence of WARC records, where each record contains the full HTTP response for one URL: status line, response headers, and body content. WARC files also contain request records showing what the crawler sent, and metadata records with crawl-specific information. A WARC record for an HTML page includes the raw HTML bytes, while a record for an image includes the raw image bytes. WARC files are the authoritative source and can be reprocessed to produce alternative text extractions if the WET extraction quality is inadequate.
WAT files (Web Archive Transformation) contain JSON metadata extracted from WARC records. Each WAT record includes HTTP headers, detected content type, page title, outgoing link structure, and language detection results. WAT files are much smaller than WARC files, typically 5-10x smaller, making them practical for metadata analysis without downloading full page content. They are useful for analyzing link graph structure, tracking domain coverage statistics, and filtering which WARC segments are worth downloading for a specific application.
WET files (WARC Encapsulated Text) contain only the plain text extracted from HTML pages, with all markup stripped. Each WET record includes the URL, timestamp, and plain text content. WET files are the most commonly used starting point for text-based NLP applications because they require no HTML parsing, have already extracted text content, and are substantially smaller than WARC files. A single monthly crawl's WET files total around 5 to 10 terabytes, compared to 70+ terabytes for the WARC files.
For language model pretraining, WET files are often the entry point. However, WET extraction quality varies significantly. Some pages produce clean, readable text, while others produce garbled outputs from malformed HTML or aggressive content extraction. Navigation menus, cookie consent dialogs, advertisement text, and JavaScript-rendered content that the simple HTML parser cannot access all end up either missing from WET files or appearing as noise. Research groups building high-quality training corpora often prefer to start from WARC files and apply their own text extraction pipeline to achieve better quality and consistency.
Accessing Common Crawl Data
Common Crawl data lives on Amazon S3 in the commoncrawl bucket. You can list available crawls and their indices via the Common Crawl index API or by browsing the S3 bucket directly. Each monthly crawl is identified by a name like CC-MAIN-2024-10 that encodes the year and approximate week number.
The Common Crawl index is a key tool for selective access. Rather than downloading entire WARC files, which are enormous, you can query the index to find specific URLs and retrieve only the relevant byte ranges using HTTP range requests. The Columnar Index format stores the index in Parquet files, allowing efficient filtered scans using tools like Apache Spark or DuckDB. You can filter by URL prefix to find all pages from a given domain, by content type to find only HTML pages, or by timestamp to find pages crawled during a specific time window, and then download only the relevant WARC byte ranges.
Let us look at what a WET record looks like in practice:
# The warcio library provides Python bindings for reading WARC/WET files
# uv pip install warcio requests
# A single WET segment file path from a Common Crawl release
# We simulate the structure by working with a minimal example
# In practice: download from s3://commoncrawl/crawl-data/CC-MAIN-2024-10/...
sample_wet_record = {
"uri": "https://en.wikipedia.org/wiki/Natural_language_processing",
"content_type": "text/plain",
"content_length": 45231,
"timestamp": "2024-03-15T14:23:11Z",
"text_preview": (
"Natural language processing (NLP) is a subfield of computer science "
"and artificial intelligence that uses machine learning to enable "
"computers to understand and communicate with human language in a "
"valuable way. NLP enables computers to read, hear, and understand "
"human language..."
),
}WET Record Structure: URI: https://en.wikipedia.org/wiki/Natural_language_processing Content-Type: text/plain Length: 45,231 bytes Timestamp: 2024-03-15T14:23:11Z Text preview: Natural language processing (NLP) is a subfield of computer science and artificial intelligence that uses machine learning to enable computers to understand and communicate with human language in a va...
Each WET record stores the URI of the crawled page alongside the extracted plain text. The content length gives us a sense of document size. For a Wikipedia article on NLP, nearly 45 KB of text reflects the depth of coverage typical for encyclopedic content. In contrast, many spam pages or thin-content sites produce only a few hundred bytes of text, a signal used by quality filters.
The practical workflow for using Common Crawl data in a training pipeline typically follows several steps. First, you identify which monthly crawl releases to use, often selecting multiple crawls across different months to ensure temporal diversity. Next, you download WAT files to analyze the metadata and identify which WARC segments contain content relevant to your goals. Then you download the WET files for those segments and apply language identification, quality filtering, and deduplication. Finally, you post-process the filtered text into the format your training pipeline expects.