Document Extraction › PDF and Other Document Formats
Language AI HandbookAll chapters Document Extraction › PDF and Other Document FormatsOne section · Sections in this chapter7 / 13 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Web crawls capture HTML, but LLM training corpora increasingly incorporate PDFs (academic papers, technical reports, books, government documents) and other formats. PDF extraction presents a distinct set of challenges that are fundamentally different from HTML extraction.
PDF Extraction Challenges
PDFs do not store text as a linear stream that mirrors the reading order. Instead, the PDF format describes text placement using absolute coordinates: "draw glyph 'H' at position (72, 720), draw glyph 'e' at position (78, 720)..." The PDF viewer assembles these glyphs into readable text visually. An extractor must reconstruct the reading order from raw positional data, which requires spatial reasoning that HTML parsing does not.
Multi-column academic papers are a common challenge. A two-column paper has text in the left column that should be read before the text at the same vertical position in the right column. A naive extractor that reads glyphs in order of their y-coordinate (top to bottom) will interleave text from both columns, producing incoherent output. The correct reading order requires first identifying column boundaries, then sorting text within each column independently.
Additional complications include:
- Scanned PDFs: documents scanned from paper contain no text at all, only images of text. These require Optical Character Recognition (OCR) before any text extraction can occur, adding significant computational cost and reducing accuracy compared to born-digital PDFs.
- Mathematical notation: equations in PDFs are often stored as sequences of specially positioned characters from mathematical symbol fonts, or as embedded images. Accurate reconstruction of mathematical formulas from raw PDF character streams is a research-grade problem; most extractors either skip equations or produce approximations.
- Headers and footers: page numbers, running chapter titles, and copyright notices appear on every page. They must be identified as repeating elements and removed, but this requires recognizing that the same text pattern appears across multiple pages.
- Tables: tabular data in PDFs loses its grid structure during extraction unless the extractor applies spatial analysis to group cells into rows and columns based on their bounding box coordinates.
- Font encoding: some PDFs use custom font encodings where the Unicode code points of characters in the PDF do not match standard Unicode assignments. The character displayed as 'A' may be stored as code point 0x41 in one font and 0x01 in another. Extractors must map through font encoding tables to recover correct characters.
PyMuPDF (fitz)
PyMuPDF (imported as fitz in Python) is among the fastest and most accurate PDF text extractors. It wraps the MuPDF library, which is also used in PDF viewers like Evince and the Chrome PDF viewer:
subprocess.run(
[sys.executable, "-m", "pip", "install", "pymupdf", "-q"],
capture_output=True,
)import fitz # PyMuPDF
# Create a minimal PDF in memory for demonstration
# In practice: doc = fitz.open("path/to/document.pdf")
# We demonstrate the API structure
def extract_text_from_pdf_bytes(pdf_bytes: bytes) -> str:
"""Extract text from PDF bytes, removing common boilerplate."""
doc = fitz.open(stream=pdf_bytes, filetype="pdf")
pages_text = []
for page_num in range(len(doc)):
page = doc[page_num]
text = page.get_text("text")
pages_text.append(text)
doc.close()
return "\n\n".join(pages_text)
# Show the API usage pattern
print("PyMuPDF API demonstrated. Key methods:")
print(" doc = fitz.open('file.pdf')")
print(" page = doc[0]")
print(" text = page.get_text('text') # plain text")
print(" blocks = page.get_text('blocks') # bounding-box blocks")
print(" words = page.get_text('words') # individual words with coords")PyMuPDF extraction modes: 'text': Raw text, preserving line breaks as in PDF 'blocks': List of (x0, y0, x1, y1, text, block_no, block_type) tuples 'words': List of (x0, y0, x1, y1, word, block_no, line_no, word_no) 'dict': Full structured representation with spans and fonts 'html': HTML reconstruction of the page layout
The "blocks" mode is particularly useful for multi-column PDFs. Each block includes its bounding box coordinates: the top-left corner and bottom-right corner. Given these bounding boxes, you can sort blocks first by column (x-coordinate) and then by vertical position within each column, reconstructing the correct reading order. A simple heuristic: if the midpoint of a block's x-range falls in the left half of the page, it belongs to the left column; if in the right half, to the right column.
The "dict" mode gives the most complete structural information, including font names, font sizes, and character-level spans. This is valuable when you want to identify headings (which typically use larger font sizes or bold weight), separate body text from captions, or detect figure and table labels.
pdfminer.six
An alternative with more fine-grained control over the extraction process is pdfminer.six, which exposes the PDF internal rendering model:
subprocess.run(
[sys.executable, "-m", "pip", "install", "pdfminer.six", "-q"],
capture_output=True,
)from pdfminer.layout import LAParams
# Demonstrate the LAParams (Layout Analysis Parameters)
laparams = LAParams(
line_overlap=0.5, # Fraction of overlap to consider lines the same
char_margin=2.0, # Max distance between chars to join into word
word_margin=0.1, # Min whitespace fraction between words
line_margin=0.5, # Max vertical distance to consider same line
boxes_flow=0.5, # Balance between column vs reading order (0=col, 1=reading)
detect_vertical=False,
all_texts=False,
)
print("LAParams configured for standard single-column documents")
print(f" char_margin: {laparams.char_margin}")
print(f" line_margin: {laparams.line_margin}")
print(f" boxes_flow: {laparams.boxes_flow}")For two-column academic papers, adjust boxes_flow=0.3 to prioritize column order. For single-column content, boxes_flow=0.5 gives good results.
The boxes_flow parameter controls the balance between two competing strategies for reconstructing reading order. A value of 0 treats each text box independently and sorts by horizontal position (column-first order). A value of 1 uses flow analysis that tries to follow the visual reading order, even across column boundaries. Values between 0 and 1 interpolate between these strategies. For academic papers, boxes_flow=0.3 typically works well; for single-column content, boxes_flow=0.5 is appropriate.
Handling Scanned Documents: OCR
For scanned PDFs and image-based documents, OCR is necessary before any text processing can begin. The two dominant open-source OCR engines are Tesseract and EasyOCR. Tesseract (developed at Hewlett-Packard and now maintained by Google) is a C++ library with Python bindings via pytesseract. EasyOCR is a more recent Python-native engine that uses deep learning models and handles a wider variety of scripts and fonts.
# pytesseract wraps the Tesseract OCR engine
# Requires: brew install tesseract (macOS) or apt install tesseract-ocr (Linux)
# subprocess.run([sys.executable, "-m", "pip", "install", "pytesseract", "-q"], capture_output=True)
# OCR pipeline example (requires Tesseract installed):
# from PIL import Image
# import pytesseract
#
# img = Image.open("scanned_page.png")
# text = pytesseract.image_to_string(img, lang="eng")
print("OCR pipeline: Image -> Tesseract -> Text")
print("Key considerations for OCR in LLM data pipelines:")
print(" - OCR confidence scores can filter low-quality extractions")
print(" - Language detection should run after OCR to identify language")
print(" - Layout analysis before OCR improves accuracy on multi-column docs")OCR introduces errors that are qualitatively different from boilerplate noise. A misrecognized character ("0" read as "O", "l" read as "1") produces text that is locally incorrect but syntactically similar to real text. These errors are difficult to detect and filter because they look plausible to statistical filters. At LLM pretraining scale, OCR errors accumulate across millions of scanned pages and can subtly degrade model accuracy on tasks involving numbers, proper names, and technical terminology.
At LLM pretraining scale, OCR is computationally expensive. Pipelines like those used to build Dolma and The Pile selectively apply OCR to PDFs that do not already contain embedded text (detected by attempting text extraction first, which is fast), avoiding unnecessary compute on the majority of documents.
Emerging Tools: Marker and Nougat
Two newer tools address specific limitations of traditional PDF extraction.
Marker is a pipeline that combines PyMuPDF for text-layer extraction with vision models for understanding complex layouts, tables, and figures. It produces Markdown output with proper heading hierarchy, preserving the logical structure of the document rather than just the raw text.
Nougat (Neural Optical Understanding for Academic Documents), developed by Meta AI, uses a vision-language model trained specifically on academic papers to convert PDFs to a structured Markdown format. It handles mathematical equations in LaTeX syntax, which is a significant improvement over existing methods. The tradeoff is that Nougat is substantially slower than traditional extractors, making it suitable for high-value academic documents where quality matters more than throughput.