Document Extraction › Building a Reliable Extraction Pipeline
Language AI HandbookAll chapters Document Extraction › Building a Reliable Extraction PipelineOne section · Sections in this chapter8 / 13 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Individual tools handle individual document types. A production extraction pipeline must orchestrate multiple tools, handle errors gracefully, track quality metrics, and make decisions about which documents are worth the extraction cost.
Document Type Detection
Before invoking any extractor, you need to know what type of document you are processing. The raw bytes provide several reliable signals. PDF files begin with the magic bytes %PDF (hex: 25 50 44 46). HTML files typically begin with a DOCTYPE declaration or an <html> tag. The URL extension and the HTTP Content-Type header are additional signals, though both can be unreliable (a server might serve HTML with a .htm extension or set the wrong content type).
A layered detection strategy is more reliable than any single signal:
from dataclasses import dataclass, field
from enum import Enum
from typing import Optional
class DocumentType(Enum):
HTML = "html"
PDF = "pdf"
UNKNOWN = "unknown"
@dataclass
class ExtractionResult:
text: Optional[str]
title: Optional[str]
language: Optional[str]
doc_type: DocumentType
char_count: int
success: bool
error: Optional[str] = None
warnings: list[str] = field(default_factory=list)
def detect_doc_type(content: bytes, url: str = "") -> DocumentType:
"""Detect document type from content signature or URL."""
if content[:4] == b"%PDF":
return DocumentType.PDF
if (
b"<html" in content[:1000].lower()
or b"<!doctype" in content[:100].lower()
):
return DocumentType.HTML
if url.endswith(".pdf"):
return DocumentType.PDF
return DocumentType.UNKNOWN
def extract_html(content: bytes) -> ExtractionResult:
"""Extract text from HTML using Trafilatura."""
try:
html_str = content.decode("utf-8", errors="replace")
text = trafilatura.extract(html_str, favor_recall=True)
metadata = extract_metadata(html_str)
title = metadata.title if metadata else None
return ExtractionResult(
text=text,
title=title,
language=None,
doc_type=DocumentType.HTML,
char_count=len(text) if text else 0,
success=text is not None and len(text) > 50,
)
except Exception as e:
return ExtractionResult(
text=None,
title=None,
language=None,
doc_type=DocumentType.HTML,
char_count=0,
success=False,
error=str(e),
)
def extract_document(content: bytes, url: str = "") -> ExtractionResult:
"""Unified document extraction dispatcher."""
doc_type = detect_doc_type(content, url)
if doc_type == DocumentType.HTML:
return extract_html(content)
# PDF extraction would follow similar pattern using fitz
return ExtractionResult(
text=None,
title=None,
language=None,
doc_type=doc_type,
char_count=0,
success=False,
error=f"Unsupported type: {doc_type.value}",
)Document type: html Extraction success: True Title: Introduction to Transformers Character count: 957 Text preview: Introduction to Transformers Published on January 15, 2024 by Jane Smith The transformer architecture, introduced in the landmark paper "Attention Is ...
Quality Filtering After Extraction
Not every extracted document is worth keeping. After extraction, a filtering pass removes documents that are too short, too repetitive, or structured in ways that suggest extraction failure rather than valid content.
Minimum length thresholds are the simplest filter. A document with fewer than 200 characters almost certainly represents a failed extraction (an empty page, a redirect, a captcha gate) rather than a legitimately short piece of content. Most production pipelines set higher thresholds: 1,000 characters or 200 words is more typical. The right threshold depends on the document types in your corpus.
Repetition filters catch a different class of problems. Certain pages consist primarily of repeated content: product listing pages where the same price and review count template repeats hundreds of times, paginated index pages where the same navigation repeats across dozens of pages, or spam pages that repeat the same keyword phrase to manipulate search engines. A simple check on the ratio of unique lines to total lines catches the most egregious cases.
def quality_filter(result: ExtractionResult) -> tuple[bool, str]:
"""Apply quality filters to an extraction result."""
if not result.success or not result.text:
return False, "extraction_failed"
text = result.text
char_count = len(text)
# Minimum content length
if char_count < 200:
return False, "too_short"
# Check for excessive repetition (lorem ipsum, test pages)
lines = [l.strip() for l in text.splitlines() if l.strip()]
if lines:
unique_line_ratio = len(set(lines)) / len(lines)
if unique_line_ratio < 0.5 and len(lines) > 5:
return False, "too_repetitive"
# Check content-to-whitespace ratio
non_space_chars = len(text.replace(" ", "").replace("\n", ""))
if non_space_chars / (char_count + 1) < 0.5:
return False, "excessive_whitespace"
return True, "passed"
# Test with our extraction result
passed, reason = quality_filter(result)Quality filter result: PASS (passed) Characters: 957 Words: 140 Unique line ratio: 1.00
More sophisticated quality filters include language identification (keeping only documents in the target language or languages), perplexity scoring against a small reference model (documents with unusually high perplexity relative to the reference language model are likely garbled or non-prose), and classifier-based quality scoring trained on human judgments of document quality.
Logging and Monitoring
A production extraction pipeline processes millions of documents per day. Without logging and monitoring, it is impossible to detect when the pipeline starts failing silently: when a site changes its HTML structure and extraction starts returning empty strings, when encoding detection starts failing for a language, or when a particular document type starts causing crashes that are silently caught by exception handlers.
The key metrics to track are: success rate (fraction of documents that return non-empty, non-error results), mean character count of successful extractions (a sudden drop signals that extraction is returning short, low-quality results), and the distribution of failure modes (which filters are catching which documents). Alerting on these metrics allows you to catch extraction quality regressions before they contaminate a large fraction of the corpus.