Web Crawling › Crawl Freshness in Depth

Understanding freshness matters operationally and epistemologically. A model does not simply know facts; it knows the snapshot of the web that existed during its training data collection. The date range of the crawl data shapes what the model considers current, recent, and historically established.

Estimating how fresh a crawl is for a given URL requires comparing the crawl timestamp to the page's last-modification time. HTTP provides two mechanisms for this. The Last-Modified response header indicates when the server believes the content was last changed. The ETag header provides a content fingerprint that changes when the content changes. Crawlers that support conditional requests can send If-Modified-Since or If-None-Match headers with a subsequent request, and the server responds with a 304 Not Modified status if the content has not changed, saving download bandwidth.

In practice, many servers do not implement these headers correctly. Pages may have Last-Modified headers that reflect the page template's modification date rather than the actual content update time. ETags may be generated randomly on each request due to server misconfiguration. Crawlers that want accurate freshness information often fall back to content change detection: hash the extracted text from the previous crawl, compare against the current hash, and flag pages as changed only when the hashes differ. This approach is accurate but requires storing a hash for every URL ever crawled, which adds significant storage overhead at web scale.

An important subtlety is that Common Crawl data is not a uniform temporal sample. Newer crawls contain more pages from recently created websites and fewer pages from sites that have since gone offline. The web has grown substantially over Common Crawl's history, so earlier crawls have sparser coverage than recent ones even for domains that have existed for many years.

This temporal non-uniformity can create training artifacts. A model trained on data from a narrow time window may underrepresent historical knowledge and overrepresent contemporary internet culture. A model trained only on 2022 crawls will have strong beliefs about what is "current" as of 2022 and little knowledge of events after that date. Mixing crawls from different time periods helps address this, and several recent training data pipelines explicitly stratify their Common Crawl samples by crawl date to ensure temporal diversity.

The model's knowledge cutoff, the date after which it has no information, arises directly from the crawl dates of its training data. When users notice a model claiming not to know about a recent event, the root cause is almost always in the crawl dates. Understanding the connection between crawl freshness and model knowledge cutoffs helps practitioners reason clearly about what their models can and cannot be expected to know.