Document Extraction › Content Extraction Approaches
Language AI HandbookAll chapters Document Extraction › Content Extraction ApproachesOne section · Sections in this chapter4 / 13 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Parsing gives you a tree. The harder problem is deciding which parts of the tree contain main content and which are boilerplate. Three main algorithmic families address this problem: rule-based extraction, density-based extraction, and machine learning classification.
Rule-Based Extraction
The simplest approach uses hand-crafted rules based on HTML semantics and structural patterns. The intuition is that content-bearing elements tend to have certain properties: they appear inside <article> or <main> tags, they consist of <p> elements with substantial text, they have high text-to-HTML ratios, and they avoid link-heavy regions.
The first step in rule-based extraction is removing known boilerplate containers before attempting to extract content from whatever remains. Navigation bars, headers, footers, sidebars, advertisements, and scripts are reliably non-content across almost all pages:
from bs4 import BeautifulSoup
def remove_boilerplate_tags(soup: BeautifulSoup) -> BeautifulSoup:
"""Remove elements that reliably contain boilerplate."""
boilerplate_selectors = [
"nav",
"header",
"footer",
"aside",
"[class*='menu']",
"[class*='sidebar']",
"[class*='ad']",
"[class*='banner']",
"[class*='cookie']",
"[class*='popup']",
"script",
"style",
"noscript",
]
for selector in boilerplate_selectors:
for tag in soup.select(selector):
tag.decompose()
return soup
sample_html = """
<html><body>
<nav><a href="/">Home</a></nav>
<article>
<h1>Understanding Neural Networks</h1>
<p>Neural networks are computational models inspired by biological neurons.</p>
<p>They consist of layers of interconnected nodes that process information.</p>
</article>
<aside class="sidebar-widget">Related articles</aside>
<footer>Privacy Policy | Terms of Service</footer>
</body></html>
"""
soup = BeautifulSoup(sample_html, "lxml")
cleaned = remove_boilerplate_tags(soup)Understanding Neural Networks Neural networks are computational models inspired by biological neurons. They consist of layers of interconnected nodes that process information.
After removing the known-boilerplate containers, the extractor looks for the best remaining content candidate. A common heuristic is to find the element with the highest density of <p> tags, on the theory that paragraphs are the natural unit of prose content. Another heuristic is to find the deepest subtree that contains the most text characters, preferring deeply nested content over shallow wrappers.
Rule-based approaches work well on sites with consistent, semantic markup and on news sites where the article is consistently inside a <article> or a <div class="article-body"> element. They fail on sites that use <div> for everything and when boilerplate classes have unpredictable or obfuscated names (some sites deliberately obfuscate class names to hinder scrapers).
The deeper problem with rule-based approaches is brittleness. Rules that work well on the 1,000 sites you tested may fail silently on the 10,000 sites you have not tested. There is no principled way to know in advance which rules will generalize. This motivates the density-based approaches that follow.
Text Density Analysis
A more robust approach exploits the observation that content regions are text-dense, while boilerplate regions are link-dense. This observation holds across a wide variety of sites and does not require knowing anything about the specific site's HTML structure or class naming conventions.
The text density of an HTML node is the ratio of characters in actual text to the total number of characters in the HTML markup for that node. Consider the arithmetic: a paragraph with 200 characters of prose text and 40 characters of wrapping tags has text density:
A navigation bar with 10 characters of link text and 300 characters of anchor tags has text density:
The contrast is dramatic and consistent across site types. Content paragraphs almost always have densities above 0.5. Navigation, advertisement, and footer regions almost always have densities below 0.2. The threshold between them provides a reliable content/boilerplate signal.
The classic algorithm implementing this idea is Content Extraction via Text-to-Link Ratio (CETR), originally proposed by YinLei Hu et al. A simplified version computes, for each block in the page:
where:
- : the number of text characters in block
- : the number of HTML tags in block
- The in the denominator prevents division by zero for tag-only nodes
Blocks with density above a threshold are classified as content; those below are classified as boilerplate.
A more detailed version also factors in the link density within each block. Navigation and footer regions are not only tag-dense, they are specifically anchor-tag-dense: a high fraction of their characters are inside <a href="..."> elements. The link density of a block is:
where:
- : characters inside anchor tags in block
- : total text characters in block
Content blocks have low link density (prose rarely consists primarily of hyperlinks). Navigation and footer blocks have high link density (they are essentially lists of links). Combining both signals gives a more robust classifier than either signal alone.
Let's implement a basic text density extractor to see these signals in practice:
import re
from bs4 import BeautifulSoup
def compute_block_features(soup: BeautifulSoup) -> list[dict]:
"""Compute text density and link density for each block-level element."""
block_tags = {
"p",
"div",
"article",
"section",
"li",
"td",
"h1",
"h2",
"h3",
}
results = []
for tag in soup.find_all(block_tags):
full_html = str(tag)
text = tag.get_text(strip=True)
char_count = len(text)
tag_count = len(re.findall(r"<[^>]+>", full_html))
# Count characters inside anchor tags
link_chars = sum(len(a.get_text(strip=True)) for a in tag.find_all("a"))
density = char_count / (tag_count + 1)
link_density = link_chars / (char_count + 1)
results.append(
{
"tag": tag.name,
"text_preview": text[:60] + "..." if len(text) > 60 else text,
"char_count": char_count,
"tag_count": tag_count,
"density": round(density, 2),
"link_density": round(link_density, 2),
}
)
return results
complex_html = """
<html><body>
<div class="nav">
<a href="/">Home</a> | <a href="/about">About</a> | <a href="/contact">Contact</a>
</div>
<article>
<h1>The Science of Deep Learning</h1>
<p>Deep learning has revolutionized computer vision, natural language processing,
and speech recognition over the past decade. The key insight is that hierarchical
representations learned from data can outperform hand-crafted features.</p>
<p>Convolutional neural networks (CNNs) learn spatial hierarchies of features,
making them particularly effective for image recognition tasks.</p>
</article>
<div class="footer-links">
<a href="/privacy">Privacy</a> | <a href="/terms">Terms</a> |
<a href="/cookies">Cookies</a> | <a href="/sitemap">Sitemap</a>
</div>
</body></html>
"""
soup3 = BeautifulSoup(complex_html, "lxml")
features = compute_block_features(soup3)Tag Density Link Density Text Preview ---------------------------------------------------------------------- div 2.00 0.84 Home|About|Contact article 45.00 0.00 The Science of Deep LearningDeep learning has revolutionized... h1 9.33 0.00 The Science of Deep Learning p 78.67 0.00 Deep learning has revolutionized computer vision, natural la... p 47.00 0.00 Convolutional neural networks (CNNs) learn spatial hierarchi... div 2.64 0.87 Privacy|Terms|Cookies|Sitemap
The output demonstrates the core principle: content paragraphs show high density and low link density, while navigation and footer divs show the opposite pattern. This signal is stable enough to generalize across sites without requiring any site-specific configuration.
The density-based approach handles the <div>-for-everything problem that defeats pure rule-based methods. Even if a navigation menu is wrapped in a generic <div> with no class names, its text density will be low because the anchor tags dominate the markup. The algorithm does not need to know that the div contains navigation; the structure of the markup reveals it implicitly.
Machine Learning-Based Extraction
The text density heuristics are hand-tuned and may not generalize perfectly across all site types. A more principled approach trains a classifier on annotated examples where human reviewers have labeled each paragraph as content or boilerplate.
The Web-Boilerplate Dataset (CLEANEVAL) provided human annotations of content and boilerplate at sentence or paragraph level, enabling supervised learning for this task. Features used in such classifiers include:
- Text density and link density (as computed above)
- Position in the document (content tends to appear near the middle; navigation at the top, footers at the bottom)
- Surrounding tag context (what parent, sibling, and child elements surround this block)
- N-gram features of the text itself (boilerplate phrases like "all rights reserved", "click here to subscribe", and "terms of service" are distinctive)
- DOM depth: navigation tends to be shallow in the tree (close to the root), while content is typically nested several levels deep
- Paragraph character count: very short paragraphs are disproportionately boilerplate
- Ratio of capital letters: navigation items and legal text often use unusual capitalization patterns
A well-trained classifier can learn non-obvious combinations of these features. For instance, a block that has medium text density but zero link density and appears directly inside an <article> element is almost certainly content, even if a density-only threshold would mark it as ambiguous. The classifier combines all available signals into a single decision.
Modern tools like Trafilatura, discussed in detail below, use a combination of rule-based signals, tree structure analysis, and machine learning heuristics, achieving strong precision-recall tradeoffs across diverse web content without requiring site-specific configuration.
The CETR Algorithm in Detail
CETR (Content Extraction via Tag Ratio) deserves a deeper look because it introduced the density-based paradigm that most modern extractors build on. The algorithm works as follows.
First, the HTML is converted to a line-by-line representation where each line corresponds to a rendered text fragment. Lines that contain only HTML tags (no visible text) are assigned a density of zero. Lines that contain substantial prose are assigned high density. This representation treats the HTML as a signal stream rather than a tree.
The density values across lines form a histogram. Content regions appear as peaks in this histogram: consecutive lines all with high density. Boilerplate regions appear as valleys. The algorithm applies a smoothing step (a moving average over adjacent lines) to reduce the impact of single noisy lines, then identifies content regions as contiguous spans where the smoothed density exceeds a threshold.
The key insight is that content appears in contiguous blocks. A paragraph consists of many consecutive text-rich lines, not isolated fragments scattered across the page. This contiguity constraint allows CETR to handle cases where individual lines have ambiguous density by looking at whether neighboring lines are also text-rich.
The weakness of CETR is that it makes a hard binary decision at a fixed threshold. Pages with unusual structures, such as academic papers with dense citations or e-commerce pages with paragraph-length product descriptions in navigation-like containers, can confuse the algorithm. More recent algorithms replace the fixed threshold with adaptive thresholds estimated per-page based on the overall density distribution.