Document Extraction › Why Document Extraction Is Hard

The web was designed for human browsers, not text extractors. HTML is a display language: its semantic structure is often weak or misleading, and publishers have no incentive to make their markup machine-readable. The result is a gap between what a page visually presents to a reader and what a naive text extractor pulls from the raw HTML bytes.

Every web page contains two kinds of text. The first is the main content: the article, the forum post, the product description, the research paper that a reader came to read. The second is boilerplate: navigation links, advertisements, footers, sidebars, related-article carousels, subscription prompts, and legal disclaimers. For training data quality, boilerplate is toxic.

The mechanism of harm is subtle. When you train a language model on text that contains thousands of repetitions of "Privacy Policy | Terms of Service | Cookie Settings", the model assigns high probability to generating exactly that kind of text in contexts where it does not belong. A model trained on a corpus with poorly removed boilerplate learns to interpolate navigation-like fragments into generated prose. It may produce text that reads naturally for a few sentences and then inexplicably inserts a list of menu items. These artifacts are difficult to detect at training time and frustrating to encounter at inference time.

Quantifying the problem: studies of Common Crawl, the most widely used web corpus for LLM pretraining, have found that after full boilerplate removal, the fraction of retained text can drop dramatically from the raw HTML byte count. A page that weighs 80 KB as HTML may yield only 3 KB of real content. The ratio varies enormously by site type. News articles have relatively high content ratios: perhaps 20-40% of the HTML bytes represent prose worth keeping. Link aggregators, forums with short posts, and shopping pages can be almost entirely boilerplate, with meaningful text occupying less than 5% of the raw bytes. At the scale of Common Crawl, which archives tens of billions of pages, even a small improvement in boilerplate removal translates to billions of higher-quality training tokens.

Semantically, HTML has tags like <article>, <main>, and <p> that should indicate content. The HTML5 specification describes <article> as "a self-contained composition in a document, page, application, or site, which is intended to be independently distributable or reusable." In theory, finding <article> tags should locate the content. In practice, many sites use <div> for everything and assign meaning only through CSS classes or JavaScript behavior. A navigation bar is often <div class="nav">. Whether that class name signals navigation depends entirely on the site's conventions, which vary infinitely across the web.

The situation is compounded by the fact that HTML semantics are rarely enforced. There is nothing stopping a developer from placing an advertisement inside an <article> tag or using <p> tags for navigation items. The semantic tags provide hints, not guarantees. An extractor that relies entirely on semantic tags will perform well on carefully constructed sites like Wikipedia or the New York Times and poorly on the long tail of hobbyist pages, small businesses, and older web content.

Additionally, HTML in the wild is frequently malformed. Studies of large web crawls consistently find that a substantial fraction of pages contain unclosed tags, incorrectly nested elements, duplicate IDs, and other violations of HTML syntax. Browsers implement extensive error-recovery heuristics to display broken HTML gracefully: they can infer where a missing closing tag should go and reconstruct a valid tree from broken markup. A parser that expects well-formed markup will fail on a significant fraction of real-world pages, silently returning empty or incorrect results.

Web pages arrive in a range of character encodings: UTF-8, Latin-1 (ISO-8859-1), Windows-1252, Shift-JIS, GB2312, Big5, and many others. UTF-8 has become dominant for new content, but a large fraction of older pages and non-English content uses other encodings. The declared encoding in the HTTP Content-Type header or the HTML <meta charset> tag is sometimes wrong, inconsistent with the actual bytes, or entirely missing.

Mishandling encoding produces mojibake: text where multi-byte sequences are decoded with the wrong charset, resulting in garbled characters. The French word "café" stored as UTF-8 (bytes: 63 61 66 c3 a9) read as Latin-1 becomes "café" with a visible artifact. At corpus scale, mojibake-corrupted text quietly poisons training data. A model trained on it will learn to associate those garbled character sequences with surrounding words, producing subtle generation artifacts for affected languages.

The correct approach is a layered encoding detection strategy: check the HTTP header first, then the HTML meta charset tag, and finally fall back to statistical encoding detection using a library like chardet or charset-normalizer. After decoding, normalize to Unicode NFC form to ensure consistent representation of characters that have multiple valid Unicode representations.

A growing fraction of the modern web delivers content via JavaScript. Single-page applications (SPAs) built with React, Vue, or Angular send a nearly empty HTML skeleton to the browser. All meaningful content is injected into the DOM by JavaScript running after page load. An HTML parser that receives the raw HTTP response from such a page sees almost nothing: a <div id="root"> or <div id="app"> with no child content.

This is increasingly common for news sites, social media platforms, and documentation portals. Standard static HTML parsing cannot handle it. The only solution is to render the page with a headless browser (Playwright, Puppeteer, or Selenium), wait for JavaScript to execute, and extract text from the rendered DOM. Headless rendering is five to fifty times slower than static parsing and consumes significantly more memory and CPU. At the scale of a web crawl, rendering every page is impractical. Production pipelines typically skip JavaScript-heavy pages entirely, or maintain a separate rendering tier for high-value domains where JavaScript content is worth the cost.