Document Extraction › Other Extraction Tools
Language AI HandbookAll chapters Document Extraction › Other Extraction ToolsOne section · Sections in this chapter6 / 13 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Trafilatura is not the only option. Several other tools occupy different points in the precision/speed/flexibility tradeoff space, and knowing their characteristics helps you choose the right tool for your specific use case.
Newspaper3k and Newspaper4k
The newspaper3k library (and its actively maintained fork newspaper4k) targets news article extraction specifically. It uses a combination of heuristics tuned for news site conventions: the assumption that the page title is in the largest heading, that the article body follows the headline, and that bylines typically appear near the top of the article.
subprocess.run(
[sys.executable, "-m", "pip", "install", "newspaper4k", "-q"],
capture_output=True,
)from newspaper import Article
from newspaper.article import ArticleDownloadState
# Demonstrate with HTML input (avoiding live network requests)
article = Article(url="https://example.com/article")
article.html = sample_page
article.is_downloaded = True
article.download_state = ArticleDownloadState.SUCCESS
article.parse()Title: Introduction to Transformers Authors: [] Text preview: Introduction to Transformers Published on January 15, 2024 by Jane Smith The transformer architecture, introduced in the landmark paper "Attention Is All You Need" (Vaswani et al., 2017), fundamenta...
Newspaper is fast to configure and extracts author and date information reliably on standard news sites. It also performs natural language processing on the article to extract summary sentences and keywords, which can be useful for building search indexes or topic-based filtering.
Its weakness is poor generalization to non-news content. Forum posts, documentation pages, e-commerce product descriptions, and academic papers do not follow the news article conventions that Newspaper's heuristics assume. For a general-purpose web corpus, Newspaper should not be the primary extractor, but it can be used as a high-quality specialized extractor for news-domain content.
Readability-lxml
The readability-lxml library is a Python port of Mozilla's Readability algorithm, which powers Firefox's "Reader Mode" feature. When you click the reader icon in Firefox, this algorithm is what strips the page down to its core content.
subprocess.run(
[sys.executable, "-m", "pip", "install", "readability-lxml", "-q"],
capture_output=True,
)from readability import Document
doc = Document(sample_page)Title: Introduction to Transformers
Extracted text preview:
Introduction to Transformers
Published on January 15, 2024 by Jane Smith
The transformer architecture, introduced in the landmark paper
"Attention Is All You Need" (Vaswani et al., 2017), fundamentally
changed natural language processing. Unlike recurrent neural networks
that process seReadability-lxml returns cleaned HTML rather than plain text. This is a meaningful distinction: the returned HTML preserves heading tags, bold and italic emphasis, table structures, and list formatting. For applications where you need structured content rather than a raw text dump, this is valuable. The algorithm works by scoring candidate blocks based on their class names, tag types, and content length, then selecting the highest-scoring subtree as the main content container.
The scoring heuristic is simple but effective: elements receive positive scores for being <div>, <p>, <td>, or <article> elements, and negative scores for class or ID names matching patterns like "comment", "meta", "footer", "nav", "sidebar", or "ad". The element with the highest cumulative score in its subtree becomes the content root.
Because it returns HTML, Readability-lxml pairs well with a secondary processing step that converts the cleaned HTML to Markdown or plain text while preserving structural signals. The markdownify library performs this conversion.
JusText
JusText implements a paragraph-level boilerplate detection algorithm that classifies each paragraph as "good" (content), "near-good", "short", or "bad" (boilerplate). It uses a combination of stop-word frequency and link density as its primary signals.
The key insight behind JusText is that stop words (high-frequency grammatical words like "the", "is", "and", "of") are distributed differently in content versus boilerplate. Ordinary prose has a natural density of stop words because language has grammatical structure. Navigation menus, copyright notices, and button labels consist primarily of content words and lack the grammatical connective tissue of real prose. JusText counts the stop-word ratio in each paragraph and uses it as a content signal.
subprocess.run(
[sys.executable, "-m", "pip", "install", "justext", "-q"],
capture_output=True,
)import justext
paragraphs = justext.justext(
sample_page.encode("utf-8"), justext.get_stoplist("English")
)Class Text Preview ------------------------------------------------------------ bad Home bad Blog short Introduction to Transformers short Published on January 15, 2024 by Jane Smith neargood The transformer architecture, introduced in the la good Self-attention allows each token to directly atten short The Attention Mechanism good At the core of the transformer is scaled dot-produ short Related Posts bad Understanding BERT bad GPT Architecture Explained bad © 2024 AI Blog. All rights reserved. bad Privacy | Terms
JusText's classification gives you more granular control than a binary content/boilerplate decision. The "near-good" class is particularly interesting: it captures paragraphs that are probably content but have some boilerplate-like characteristics (perhaps a short paragraph that happens to have low stop-word density). You can choose to include "near-good" paragraphs for recall-heavy pipelines or restrict to "good" only for high-precision pipelines.
JusText's reliance on stop words also means it handles multilingual content differently from English-centric heuristics. Stop word lists exist for many languages, and the algorithm's behavior is consistent across languages as long as the appropriate stop word list is used. This makes JusText a good choice for multilingual corpora where English-specific heuristics may fail.
Comparing the Tools
Each tool reflects a different design philosophy and set of tradeoffs:
- Trafilatura is the most general-purpose and is best suited for processing diverse web corpora at scale. Its combination of rule-based, density-based, and machine-learning signals makes it robust across site types.
- Newspaper4k is the best choice for news content specifically. Its domain-specific heuristics achieve excellent accuracy on news sites while being simpler to configure than Trafilatura.
- Readability-lxml is best when you need structured output (preserved HTML with headings, lists, and tables) rather than plain text, and when your content is articles or documentation.
- JusText is best for multilingual corpora and when granular classification (good/near-good/bad) is useful for your downstream filtering logic.