Web Crawling › Limitations and Practical Challenges

Web crawling at scale poses both engineering and sociotechnical challenges. Several important limitations shape what crawled data can and cannot represent.

The web is not uniformly accessible. Content behind login walls, paywalls, or JavaScript-rendered single-page applications is largely invisible to simple crawlers. Academic journals, enterprise intranets, subscription newsletters, and private social media posts collectively represent an enormous body of knowledge that web crawling cannot access. Common Crawl's text is therefore biased toward publicly accessible content, which skews toward English-language, commercially oriented, and technically accessible writing.

Dynamic content generation poses a separate challenge. A growing fraction of web pages are rendered entirely client-side using JavaScript frameworks. A basic HTTP crawler receives only the empty HTML skeleton, missing all the content that JavaScript would populate. Crawling JavaScript-rendered pages requires a full browser automation layer using tools like Playwright or Puppeteer, which adds significant complexity and resource cost. Running a headless browser for every page is 10 to 50 times slower and more resource-intensive than simple HTTP fetching. For this reason, most large-scale crawls still use HTTP-only fetching and accept the coverage gap for JavaScript-heavy sites.

The language distribution of the web also creates bias. English dominates the web in terms of both page count and content density, but the actual population of potential language AI users is globally distributed. Common Crawl contains content in dozens of languages, but English pages vastly outnumber pages in any other language. Models trained on unfiltered Common Crawl data inherit this English-language dominance. Building multilingual training data requires deliberate effort: explicitly identifying and proportionally including content from underrepresented languages rather than relying on raw crawl proportions.

Without domain capping, link-following crawlers naturally accumulate enormous numbers of pages from a few highly interconnected domains. A crawler that follows all links would eventually download billions of Reddit comments and Stack Overflow answers, which, while potentially useful, would swamp other content types. This concentration effect arises from the web's power-law link structure: the most popular domains receive many times more inbound links than less popular ones, so a link-following crawler visits them many times more often.

Production data pipelines typically apply domain caps, limiting any single domain to a fixed fraction of the total corpus, to ensure diversity. If Reddit alone contributes 20% of a training corpus, the model will be heavily biased toward casual conversational writing and the specific cultural norms of Reddit communities. A cap of 1% per domain across thousands of domains produces a much more balanced representation of writing styles and topics.

Copyright law creates significant uncertainty around web crawling for training data purposes. While robots.txt provides a social contract for access permission, it does not confer copyright permission. Content creators increasingly argue that training language models on their writing without compensation constitutes copyright infringement. Several lawsuits in the United States and Europe have challenged major AI companies on exactly these grounds, with ongoing legal proceedings that have not yet produced clear settled law.

The tension remains unresolved. Open research into language AI depends on large-scale crawled data. But the people who created that data, journalists, authors, coders, educators, often received no compensation and gave no explicit consent. Responsible use of crawled data should at minimum respect robots.txt, provide opt-out mechanisms, and acknowledge the human labor that produced the training corpus. Some organizations have begun offering opt-out registries where content creators can request removal from training datasets, though the practical effectiveness of these mechanisms remains limited.

The situation is evolving rapidly. The European Union's AI Act and related regulations are establishing new requirements around training data transparency. Some jurisdictions are exploring data rights frameworks that would give content creators enforceable claims over uses of their work. Anyone building a production crawling pipeline should monitor legal developments in their target jurisdictions and consult legal counsel before deploying at scale.

The web is not a curated library; it is the sum of everything humans have chosen to publish. This includes rigorous scholarship alongside spam, misinformation, hate speech, and low-effort content. A crawl that follows links aggressively will collect substantial quantities of all of these. Spam farms generate thousands of keyword-stuffed pages designed to manipulate search rankings rather than inform readers. Content mills produce formulaic articles that repeat the same information in slightly varied phrasing, contributing enormous volume with minimal unique information. Deliberately misleading content, conspiracy theories, and factually incorrect claims appear throughout the web at substantial scale.

The downstream filtering pipeline is as important as the crawl itself. A thoughtfully filtered medium-sized crawl often produces better training data than a massive unfiltered one. We discuss quality filtering, toxicity filtering, and deduplication in the following chapters in this section. For now, the key insight is that web crawling produces raw material that requires substantial refinement before it is suitable as training data. The crawl is the beginning of the pipeline, not the end.