Document Extraction › Document Extraction

Training a language model at scale starts long before any gradient update. Before text can enter a training pipeline, it must first be extracted from the raw formats in which it exists: HTML pages downloaded by web crawlers, PDFs, Word documents, and other structured formats. The process of extracting clean, readable text from these formats is called document extraction, and it is one of the most consequential steps in the entire data curation pipeline. Get this step wrong and every downstream stage, from tokenization to model training, inherits the damage. Get it right and you create a foundation of clean, coherent prose that a model can learn from.

Building on the Web Crawling chapter, which explained how to collect raw HTML at scale, this chapter focuses on the next step: converting that raw HTML (and other document formats) into clean text suitable for model training. The challenge is harder than it sounds. A typical web page bundles navigation menus, sidebars, cookie consent banners, footer links, advertisements, and tracking scripts alongside the actual article or blog post you care about. Naive extraction pulls everything indiscriminately. The result is text poisoned with "Click here to subscribe" and "Copyright 2023 XYZ Corp" repeated thousands of times, patterns a language model will dutifully memorize and regurgitate at the worst possible moments.

This chapter covers the technical mechanics of HTML parsing, the problem of boilerplate removal, the main algorithmic approaches to content extraction, and production-ready tools like Trafilatura that implement these ideas. We close by looking at PDF extraction, a separate and harder problem, and by examining how to build a robust multi-format extraction pipeline with proper quality filtering.