Document Extraction › Limitations and Practical Considerations

Document extraction is reliable on well-structured HTML and standard document formats, but it has limits that practitioners must account for at scale.

The web contains an enormous variety of specialized formats: JavaScript-rendered single-page applications (SPAs) where content only appears after JavaScript executes, Flash-based content (largely defunct but still present in archived crawls), XML feeds, custom CMS markup, and more. Standard HTML extractors cannot process SPAs because they receive essentially blank HTML before JavaScript runs. Handling SPAs requires a headless browser like Playwright or Puppeteer, which is orders of magnitude slower and more resource-intensive than static parsing.

For LLM pretraining corpora, SPAs are often simply excluded: the added infrastructure cost is not justified when billions of static HTML pages are available. But for specialized domains (financial dashboards, real-time data applications, social media platforms), SPA rendering becomes necessary if those domains are important to the target use case.

The long tail also includes content that is technically accessible as HTML but deliberately obfuscated to prevent extraction: encrypted or encoded content bodies, dynamically assembled content via <script> injection, and anti-scraping techniques that detect non-browser user agents and return empty or misleading content. These techniques are increasingly common on sites that are concerned about their content being used for AI training.

No extractor achieves uniform accuracy across all site types. Trafilatura's creators report precision and recall figures in the 85-95% range on news corpora, but accuracy drops significantly on forums, Q&A sites, and non-English pages with unusual character distributions. The benchmark numbers reported in research papers are measured on curated test sets that may not represent the distribution of content in a real web crawl.

Pipeline builders should evaluate their chosen extractor on a representative sample of their target corpus before committing to it, rather than assuming published benchmarks generalize. A practical approach is to manually inspect 100-200 randomly sampled extraction results, noting cases where content was dropped (false negatives) or boilerplate was retained (false positives), then adjusting extractor settings or adding post-processing steps to address the most common failure modes.

Even with careful encoding detection, at corpus scale you will encounter documents where the declared encoding is wrong, the encoding is a rare dialect with no detection support, or the file is simply corrupted. Standard practice is to apply a final UTF-8 encoding pass with error replacement (errors='replace'), which substitutes the Unicode replacement character (U+FFFD, displayed as ) for undecodable bytes. Documents with high replacement-character density can then be filtered out.

The threshold for filtering depends on the target languages in your corpus. Scripts that encode characters as multi-byte sequences in UTF-8 (Chinese, Japanese, Korean, Arabic, and many others) will naturally have higher byte counts per character, but the replacement-character density should still be near zero for correctly encoded documents. A replacement rate above 1-2% suggests encoding problems worth filtering on.

For model pretraining, recall matters more than it might for other applications. Every valuable paragraph that an extractor misclassifies as boilerplate is permanently lost from the training data. Practitioners frequently find that a recall-biased extractor combined with aggressive downstream quality filtering outperforms a precision-biased extractor on final model quality, because the quality filtering can remove noise that slipped through, but cannot recover content that was never extracted.

The right balance depends on the downstream pipeline. If you have a robust multi-stage quality pipeline (deduplication, perplexity filtering, classifier-based quality scoring), favor recall at the extraction stage. If extraction is your only quality gate, favor precision.

This tradeoff has an asymmetry worth emphasizing: boilerplate that passes through extraction will be seen repeatedly across many documents (a given footer template might appear on every page of a large site), and the repetition signal allows downstream deduplication to identify and remove it. But content that is filtered at extraction time disappears from the corpus entirely and permanently. Erring on the side of inclusion at the extraction stage, with careful filtering downstream, is generally the safer strategy.

Document extraction from web crawls raises questions about copyright and terms of service that the technical literature often underweights. Many websites prohibit automated scraping in their terms of service. The legality of using scraped content for AI training is an area of active litigation in multiple jurisdictions, with outcomes that will shape industry practice for years to come. The robots.txt crawl politeness standard described in the Web Crawling chapter applies equally here: content that was crawled against a site's expressed wishes remains ethically questionable even after successful extraction.

The appropriate response to this legal uncertainty varies by organization and use case. Academic research pipelines often operate under fair use or equivalent doctrines. Commercial AI training at scale faces more scrutiny. The safest approaches involve obtaining explicit permission, using licensed datasets (such as Common Crawl, which provides data under its terms of use), or restricting training data to content with permissive licensing.

Current text extraction pipelines discard everything that is not text: images, charts, diagrams, and tables. For a purely text-focused language model, this is acceptable. But much of the knowledge in academic papers, technical documentation, and even news articles is conveyed through visual elements. A scientific paper describing a new algorithm might have a diagram that makes the algorithm immediately clear but whose meaning is lost when the paper is reduced to its text.

Multimodal training, where models learn from text and images together, requires preserving and linking these visual elements during extraction. This is an active area of development: tools like Nougat and Marker are beginning to handle figures and tables in PDF extraction, and some HTML extractors preserve <img> tags with their alt text to provide at least a textual representation of visual content.