Document Extraction › HTML Parsing Fundamentals
Language AI HandbookAll chapters Document Extraction › HTML Parsing FundamentalsOne section · Sections in this chapter3 / 13 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Before any content extraction algorithm can run, the raw HTML bytes must be parsed into a structured representation that the algorithm can traverse and analyze. Two dominant approaches exist: tree-based parsing and streaming parsing.
DOM Tree Parsing
Tree-based parsers convert the entire HTML document into a Document Object Model (DOM): a hierarchical tree of nodes where each element, attribute, and text fragment is a node. The root of the tree is the <html> element. Its children are <head> and <body>. The body contains all the visible page elements as nested descendants.
Once the tree is built, you can traverse it with XPath queries, CSS selectors, or recursive traversals. You can walk from any node to its parent, children, or siblings. You can select all <p> elements within an <article> element, or all elements whose class attribute contains a particular substring. This rich navigation model is what makes DOM-based extraction powerful.
The primary Python library for DOM-based parsing is Beautiful Soup, which wraps underlying parsers (lxml or the built-in html.parser) and provides a high-level API for tree navigation:
from bs4 import BeautifulSoup
html = """
<html>
<head><title>Example</title></head>
<body>
<nav><a href="/">Home</a><a href="/about">About</a></nav>
<main>
<h1>Article Title</h1>
<p>This is the first paragraph of the article.</p>
<p>This is the second paragraph with more content.</p>
</main>
<footer>Copyright 2024. All rights reserved.</footer>
</body>
</html>
"""
soup = BeautifulSoup(html, "lxml")Title: Example Main content: Article Title This is the first paragraph of the article. This is the second paragraph with more content.
The get_text() method extracts all text within a node, joining inline elements and stripping tags. The separator argument controls how block-level elements are joined, which matters when you need to preserve paragraph breaks for downstream processing.
Beautiful Soup's tree-building step involves two phases: lexing (breaking the HTML byte stream into tokens: open tags, close tags, attributes, text nodes, and comments) and parsing (assembling the token stream into a tree according to HTML grammar rules). The lxml backend handles this more efficiently than Python's built-in html.parser and applies browser-compatible error recovery for malformed HTML.
lxml and XPath
For high-performance parsing at scale, lxml provides direct bindings to the libxml2 C library. It parses faster than Python-native parsers and supports XPath, a powerful query language for selecting nodes in a tree:
from lxml import html as lhtml
tree = lhtml.fromstring(html.encode("utf-8"))
# XPath to select all paragraph text within main
paragraphs = tree.xpath("//main//p/text()")Paragraphs found: - This is the first paragraph of the article. - This is the second paragraph with more content.
XPath expressions describe paths through the tree: //main//p/text() means "find all <p> elements anywhere inside a <main> element, and return their text content." XPath supports predicates (filtering by attribute values), positional indexing, and aggregate functions, making it possible to write highly targeted queries.
XPath is powerful when you know the structure in advance: for example, parsing a specific news site with a stable template. It becomes unwieldy when dealing with arbitrary pages at crawl scale, where the structure is unpredictable. An XPath rule that works perfectly on one site may extract nothing or the wrong content on a structurally different site.
Handling Malformed HTML
The lxml HTML parser implements the same error recovery heuristics as browsers, making it robust to the broken markup common in real web data. Beautiful Soup's lxml backend inherits this robustness:
broken_html = "<p>Unclosed paragraph <b>bold text <p>Second paragraph"
soup2 = BeautifulSoup(broken_html, "lxml")<html>
<body>
<p>
Unclosed paragraph
<b>
bold text
</b>
</p>
<p>
Second paragraph
</p>
</body>
</html>The parser infers closing tags and produces a valid tree from the broken input. In this case, lxml recognizes that <p> elements cannot be nested (in the HTML content model, a <p> element is implicitly closed when another block-level element begins), so it creates two sibling <p> elements, each properly closed. This error recovery is essential for processing real-world web data: a pipeline that crashes or returns empty results on malformed HTML will lose a significant fraction of crawled pages.
The Tree as a Data Structure
Understanding the DOM tree as a data structure clarifies why DOM-based parsing is so useful for extraction algorithms. Each node in the tree has a type (element, text, comment, CDATA), a tag name (for element nodes), a set of attributes, and a list of children. The tree preserves the full containment hierarchy of the page.
This hierarchy is the key signal for content extraction. If a block of text is contained within a <nav> element, it is almost certainly boilerplate, regardless of its textual content. If it is contained within an <article> element that is itself a child of <main>, it is almost certainly content. The extraction problem reduces, in part, to learning which subtrees of the DOM contain content and which contain boilerplate, using the tree structure itself as a feature.