Document Extraction › Trafilatura: Industrial-Strength Extraction
Language AI HandbookAll chapters Document Extraction › Trafilatura: Industrial-Strength ExtractionOne section · Sections in this chapter5 / 13 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Trafilatura is an open-source Python library specifically designed for web content extraction at scale. It was developed with LLM pretraining data quality in mind and has become a standard tool in the field. It handles HTML parsing, boilerplate removal, encoding detection, and output formatting in a single, well-tested pipeline.
How Trafilatura Works
Trafilatura's extraction pipeline has several stages that combine multiple signals to make robust content/boilerplate decisions.
The first stage attempts to locate the content candidate using XPath rules targeting semantic HTML elements like <article>, <main>, <section>, and high-density <p>-containing <div> elements. Trafilatura also uses class and ID heuristics to identify content regions even when semantic tags are absent: class names containing "article", "content", "post-body", or "entry-content" are strong positive signals. Class names containing "nav", "menu", "sidebar", "footer", "ad", "banner", or "cookie" are strong negative signals.
The second stage computes tree-level features for candidate blocks, including text density, link density, and paragraph length distribution. Blocks are scored and filtered based on these features. Short blocks with high link density are discarded. Blocks consisting primarily of boilerplate indicators (short copyright notices, "read more" links, metadata fragments) are also removed.
The third stage reassembles the surviving blocks into a coherent document, preserving heading structure and paragraph breaks. Trafilatura is careful to maintain the logical reading order of the original document rather than simply concatenating extracted blocks. It can output plain text, Markdown, or structured XML depending on the use case.
Trafilatura also handles encoding detection automatically, using a cascade of strategies: HTTP headers (if metadata is provided), HTML meta tags, and statistical detection as a fallback. It normalizes all output to UTF-8.
Basic Usage
import subprocess
import sys
# Install trafilatura if not present
subprocess.run(
[sys.executable, "-m", "pip", "install", "trafilatura", "-q"],
capture_output=True,
)import trafilatura
# Simulate fetching and extracting a page
# In practice: downloaded_html = trafilatura.fetch_url(url)
sample_page = """
<!DOCTYPE html>
<html>
<head><meta charset="utf-8"><title>Introduction to Transformers</title></head>
<body>
<header>
<nav><ul><li><a href="/">Home</a></li><li><a href="/blog">Blog</a></li></ul></nav>
</header>
<main>
<article>
<h1>Introduction to Transformers</h1>
<p>Published on January 15, 2024 by Jane Smith</p>
<p>The transformer architecture, introduced in the landmark paper
"Attention Is All You Need" (Vaswani et al., 2017), fundamentally
changed natural language processing. Unlike recurrent neural networks
that process sequences step by step, transformers process all tokens
in parallel using a mechanism called self-attention.</p>
<p>Self-attention allows each token to directly attend to every other
token in the sequence, regardless of distance. This enables transformers
to capture long-range dependencies that RNNs struggle with, and to
train much more efficiently on modern hardware due to parallelism.</p>
<h2>The Attention Mechanism</h2>
<p>At the core of the transformer is scaled dot-product attention.
For each query vector, attention computes dot products against all
key vectors, scales by the square root of the dimension, applies
softmax to obtain weights, and takes a weighted sum of value vectors.</p>
</article>
</main>
<aside>
<h3>Related Posts</h3>
<ul>
<li><a href="/bert">Understanding BERT</a></li>
<li><a href="/gpt">GPT Architecture Explained</a></li>
</ul>
</aside>
<footer>
<p>© 2024 AI Blog. All rights reserved.</p>
<nav><a href="/privacy">Privacy</a> | <a href="/terms">Terms</a></nav>
</footer>
</body>
</html>
"""
extracted = trafilatura.extract(sample_page)
extracted_with_meta = trafilatura.extract(
sample_page,
include_comments=False,
include_tables=True,
favor_recall=False,
output_format="markdown",
)=== Plain text extraction === Introduction to Transformers Published on January 15, 2024 by Jane Smith The transformer architecture, introduced in the landmark paper "Attention Is All You Need" (Vaswani et al., 2017), fundamentally changed natural language processing. Unlike recurrent neural networks that process sequences step by step, transformers process all tokens in parallel using a mechanism called self-attention. Self-attention allows each token to directly attend to every other token in the sequence, regardless of distance. This enables transformers to capture long-range dependencies that RNNs struggle with, and to train much more efficiently on modern hardware due to parallelism. The Attention Mechanism At the core of the transformer is scaled dot-product attention. For each query vector, attention computes dot products against all key vectors, scales by the square root of the dimension, applies softmax to obtain weights, and takes a weighted sum of value vectors. === Markdown extraction === # Introduction to Transformers Published on January 15, 2024 by Jane Smith The transformer architecture, introduced in the landmark paper "Attention Is All You Need" (Vaswani et al., 2017), fundamentally changed natural language processing. Unlike recurrent neural networks that process sequences step by step, transformers process all tokens in parallel using a mechanism called self-attention. Self-attention allows each token to directly attend to every other token in the sequence, regardless of distance. This enables transformers to capture long-range dependencies that RNNs struggle with, and to train much more efficiently on modern hardware due to parallelism. ## The Attention Mechanism At the core of the transformer is scaled dot-product attention. For each query vector, attention computes dot products against all key vectors, scales by the square root of the dimension, applies softmax to obtain weights, and takes a weighted sum of value vectors.
Trafilatura correctly isolates the article body, discarding the navigation, sidebar, and footer. The Markdown output preserves the heading hierarchy, which is valuable when the extracted text will be used in a retrieval system or a training corpus where structural signals matter.
Metadata Extraction
Beyond the main text, Trafilatura can extract structured metadata from pages: titles, author names, publication dates, and canonical URLs. This metadata is valuable for filtering and provenance tracking.
Publication dates allow you to filter content by recency, which matters when training data quality depends on the time period. Author information can help identify and deduplicate content from the same source appearing under different URLs. Canonical URLs help resolve the common situation where the same article is accessible at multiple URLs (with and without trailing slashes, with different UTM parameters, syndicated to partner sites).
metadata = extract_metadata(sample_page)Title: Introduction to Transformers Author: None Date: 2024-01-15 URL: None
Trafilatura extracts metadata from multiple sources in the page: the <title> tag, <meta> tags with properties like og:title and og:author, JSON-LD structured data blocks (a machine-readable format increasingly used by modern websites), and microdata annotations. This multi-source approach increases robustness: if the <title> tag is missing or generic, the og:title property often contains the real article title.
Extraction Quality Settings
Trafilatura exposes a favor_recall versus precision tradeoff that you can control explicitly:
- When
favor_recall=True, Trafilatura includes more candidate text, accepting some boilerplate in order to avoid discarding real content. This is preferable when building training corpora where missing content is more costly than having some noise, because downstream quality filtering can remove boilerplate that slips through, but cannot recover content that was never extracted. - When
favor_recall=False(the default), the extractor is more conservative, keeping only high-confidence content blocks. This yields cleaner output but may discard short but valid paragraphs, particularly on pages with sparse content.
short_content_page = """
<html><body>
<nav><a href="/">Home</a><a href="/news">News</a></nav>
<article>
<h1>Quick Update</h1>
<p>We released version 2.0 today.</p>
</article>
<footer>Contact us at info@example.com</footer>
</body></html>
"""
precise = trafilatura.extract(short_content_page, favor_recall=False)
recalled = trafilatura.extract(short_content_page, favor_recall=True)Precision mode: 'Home\nNews\nQuick Update\nWe released version 2.0 today.' Recall mode: 'Home\nNews\nQuick Update\nWe released version 2.0 today.'
For LLM pretraining, recall-favoring settings are often preferable because the downstream deduplication and quality filtering steps can remove remaining noise, whereas content that is never extracted cannot be recovered. The optimal choice depends on how sophisticated your downstream filtering is. A pipeline with perplexity-based quality scoring and n-gram deduplication can tolerate more extraction noise than a pipeline that relies on extraction quality alone.
Performance at Scale
Trafilatura is designed to handle the throughput demands of processing a full web crawl. On a single CPU core, it can extract text from thousands of pages per second when processing pre-downloaded HTML. The library includes built-in parallelism support for batch processing:
# Batch processing pattern for large corpora
# trafilatura.process_xmltei() handles batches efficiently
# For maximum throughput, process HTML in parallel using multiprocessing
from typing import Optional
def safe_extract(html: str) -> Optional[str]:
"""Extract text from HTML with error handling."""
try:
return trafilatura.extract(html, favor_recall=True)
except Exception:
return None
# Example of batch processing structure
html_documents = [sample_page, short_content_page]
results = list(map(safe_extract, html_documents))Processed: 2 documents Successful extractions: 2 Total characters extracted: 1010
At LLM pretraining scale, where you might process hundreds of terabytes of HTML, the throughput characteristics of your extraction library matter enormously. Trafilatura's C-library backend (via lxml) keeps per-page extraction time in the single-millisecond range for typical pages, making it feasible to process billions of pages on a cluster in reasonable time.