Document Extraction › Summary

Document extraction bridges raw web data and usable training text. The key ideas to carry forward:

• Boilerplate removal is as important as content extraction. A naive text dump of HTML is polluted with navigation, advertising, and legal text that degrades model training by introducing repetitive, context-free fragments.
• Text density and link density are the core signals that most algorithms rely on. Content regions are character-dense and link-sparse; boilerplate regions are the opposite. This signal holds across diverse site types without requiring site-specific configuration.
• The DOM tree is the central data structure for HTML extraction. Understanding the tree enables principled feature engineering and makes it possible to exploit the containment hierarchy as a content signal.
• Trafilatura is the state-of-the-practice tool for HTML extraction at LLM scale, giving strong precision-recall balance, metadata extraction, and reliable encoding handling. Its recall-favoring mode is appropriate for pretraining pipelines with downstream quality filtering.
• PDF extraction is a separate and harder problem, requiring spatial reconstruction of reading order from absolute glyph coordinates. PyMuPDF handles most cases well; multi-column layouts require additional spatial reasoning. Scanned PDFs require OCR, which is computationally expensive and introduces a different class of errors.
• Favor recall at extraction time when a downstream quality pipeline will further filter the corpus. Missed content cannot be recovered; boilerplate that slips through can be removed by deduplication.
• Evaluate on your specific data. Published benchmark numbers do not substitute for empirical testing on representative samples of your target crawl. Extraction quality varies significantly across site types and languages.
• The legal and ethical rules around web extraction for AI training is actively evolving. Be aware of the terms of service of sources you extract from and the licensing of datasets you use.

The next chapters in this section cover deduplication and quality filtering: the downstream stages that refine the text this extraction pipeline produces into a training corpus of sufficient quality for language model pretraining.