Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Document Extraction › Comparing Extraction Tools

Michael BrenndoerferJanuary 9, 20263 min read

Language AI HandbookAll chapters

Part 20 of 67Data Curation

Chapter 140Document Extraction › Comparing Extraction Tools

Reading section 9 of 13 · Comparing Extraction Tools · 3 minSection 9 of 13 · 3 min

Section 9 of 13 · 3 min read

Document Extraction › Comparing Extraction ToolsOne section · Sections in this chapter9 / 13 ↓

Sections in this chapter13

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

The right tool depends on the use case. Evaluating tools on the same document reveals how their approaches differ in practice:

In[43]:

Code

test_document = """
<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <title>Backpropagation Through Time: A Complete Guide</title>
</head>
<body>
<header>
  <nav class="main-nav">
    <a href="/">Machine Learning Blog</a>
    <a href="/tutorials">Tutorials</a>
    <a href="/subscribe">Subscribe</a>
  </nav>
</header>
<div class="breadcrumb">Home > Tutorials > Deep Learning > BPTT</div>
<main class="content">
  <article>
    <h1>Backpropagation Through Time: A Complete Guide</h1>
    <div class="meta">By Alex Johnson | March 15, 2024 | 8 min read</div>
    <p>Backpropagation through time (BPTT) is the algorithm used to train
    recurrent neural networks (RNNs). It extends the standard backpropagation
    algorithm to handle sequences by "unrolling" the network across time steps
    and applying the chain rule across the resulting computation graph.</p>
    <p>Understanding BPTT is essential for grasping why RNNs are difficult to
    train on long sequences. The gradient must flow backward through every time
    step, and repeated multiplication of gradient matrices causes either
    vanishing or exploding gradients.</p>
    <h2>The Vanishing Gradient Problem</h2>
    <p>When gradients flow through many time steps, repeated matrix multiplication
    causes them to shrink exponentially. A gradient that starts at 1.0 multiplied
    by 0.9 at each of 50 steps becomes less than 0.005. The weights at early time
    steps receive nearly zero gradient signal and fail to learn long-range dependencies.</p>
  </article>
</main>
<aside class="sidebar">
  <div class="widget">
    <h4>Popular Articles</h4>
    <ul>
      <li><a href="/lstm">LSTM Networks Explained</a></li>
      <li><a href="/gru">GRU: Gated Recurrent Units</a></li>
      <li><a href="/transformers">Why Transformers Beat RNNs</a></li>
    </ul>
  </div>
  <div class="ad-banner">Advertisement</div>
</aside>
<footer>
  <p>© 2024 ML Blog. Privacy Policy | Cookie Settings | Contact</p>
</footer>
</body>
</html>
"""

# Extract using each tool
trafilatura_result = trafilatura.extract(test_document) or ""
bs4_result = remove_boilerplate_tags(
    BeautifulSoup(test_document, "lxml")
).get_text(separator="\n", strip=True)

Out[44]:

Console

Trafilatura:
  Words extracted: 136
  Boilerplate markers retained: 0
  Preview: Backpropagation Through Time: A Complete Guide Backpropagation through time (BPTT) is the algorithm used to train recurr

BS4 + Rules:
  Words extracted: 153
  Boilerplate markers retained: 0
  Preview: Backpropagation Through Time: A Complete Guide Backpropagation Through Time: A Complete Guide By Alex Johnson | March 15

The comparison reveals that rule-based removal using Beautiful Soup, while straightforward to implement, often leaves residual boilerplate that Trafilatura's more sophisticated pipeline eliminates. The rule-based approach removes known boilerplate containers but cannot identify boilerplate that appears inside ambiguous <div> elements or that uses non-standard class names.