Multimodal Applications › Code Implementation
Language AI HandbookAll chapters Multimodal Applications › Code ImplementationOne section · Sections in this chapter6 / 10 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
This section builds a practical document understanding pipeline: we will load a document image, run layout-aware OCR to extract text with bounding boxes, and implement a simple document question answering system that uses spatial context to answer queries about document structure. We will also implement the video frame sampling comparison that shows the key tradeoffs in temporal video analysis.
We start by installing the required packages and setting up the processing environment.
# uv pip install pillow pytesseract transformers torch numpy matplotlibimport warnings
warnings.filterwarnings("ignore")Next, we create a synthetic document image that simulates the structure of a real business document, with labeled fields and values arranged in a grid layout. Using a synthetic document lets us control exactly what the ground truth is, so we can evaluate whether our layout-aware extraction correctly identifies values for each field.
from PIL import Image, ImageDraw
def create_synthetic_invoice():
"""Create a synthetic invoice image with layout structure."""
width, height = 600, 800
img = Image.new("RGB", (width, height), "white")
draw = ImageDraw.Draw(img)
# Title section
draw.rectangle([20, 20, 580, 70], outline="black", width=2, fill="#f0f0f0")
draw.text((30, 35), "INVOICE", fill="black")
draw.text((400, 35), "Invoice #: INV-2024-001", fill="black")
# Header fields
fields = [
("Bill To:", "Acme Corporation"),
("Address:", "123 Main Street, NY 10001"),
("Date:", "2024-01-15"),
("Due Date:", "2024-02-15"),
("Payment Terms:", "Net 30"),
]
y_offset = 100
for label, value in fields:
draw.text((30, y_offset), label, fill="black")
draw.text((180, y_offset), value, fill="#333333")
y_offset += 30
# Table header
draw.rectangle([20, 260, 580, 300], outline="black", fill="#4472c4")
draw.text((30, 270), "Description", fill="white")
draw.text((250, 270), "Qty", fill="white")
draw.text((340, 270), "Unit Price", fill="white")
draw.text((460, 270), "Amount", fill="white")
# Line items
items = [
("Software License", "1", "$2,500.00", "$2,500.00"),
("Implementation Services", "40 hrs", "$150.00", "$6,000.00"),
("Training Workshop", "2 days", "$800.00", "$1,600.00"),
("Annual Support Plan", "1", "$1,200.00", "$1,200.00"),
]
y_offset = 310
for i, (desc, qty, price, amount) in enumerate(items):
fill = "#f8f8f8" if i % 2 == 0 else "white"
draw.rectangle(
[20, y_offset, 580, y_offset + 35], fill=fill, outline="#dddddd"
)
draw.text((30, y_offset + 10), desc, fill="black")
draw.text((255, y_offset + 10), qty, fill="black")
draw.text((345, y_offset + 10), price, fill="black")
draw.text((465, y_offset + 10), amount, fill="black")
y_offset += 35
# Totals section
totals_y = 470
draw.rectangle(
[350, totals_y, 580, totals_y + 30], fill="#f0f0f0", outline="#cccccc"
)
draw.text((360, totals_y + 8), "Subtotal:", fill="black")
draw.text((465, totals_y + 8), "$11,300.00", fill="black")
draw.rectangle(
[350, totals_y + 30, 580, totals_y + 60],
fill="#f0f0f0",
outline="#cccccc",
)
draw.text((360, totals_y + 38), "Tax (8.5%):", fill="black")
draw.text((465, totals_y + 38), "$960.50", fill="black")
draw.rectangle(
[350, totals_y + 60, 580, totals_y + 90],
fill="#4472c4",
outline="#4472c4",
)
draw.text((360, totals_y + 68), "TOTAL DUE:", fill="white")
draw.text((465, totals_y + 68), "$12,260.50", fill="white")
return img
invoice_img = create_synthetic_invoice()Invoice image size: 600 x 800 pixels Document contains: header, address fields, line item table, and totals section
Now we implement a bounding-box-aware token extractor that simulates what layout-aware models receive as input. In a production system, this would call Tesseract OCR or a specialized document parser such as DocTR or PaddleOCR. The key output includes the recognized text and the normalized bounding box coordinates that encode each token's spatial position.
def extract_layout_tokens(image):
"""
Extract text tokens with normalized bounding box coordinates.
Simulates output from a layout-aware OCR system.
Returns list of dicts with 'text' and 'bbox' (normalized 0-1000).
"""
width, height = image.size
# Simulated OCR output with bounding boxes (what Tesseract or DocTR would produce)
raw_tokens = [
# Header
("INVOICE", 30, 35, 180, 55),
("Invoice", 400, 35, 460, 55),
("#:", 462, 35, 480, 55),
("INV-2024-001", 482, 35, 580, 55),
# Address fields
("Bill", 30, 100, 70, 115),
("To:", 72, 100, 95, 115),
("Acme", 180, 100, 220, 115),
("Corporation", 222, 100, 310, 115),
("Date:", 30, 160, 80, 175),
("2024-01-15", 180, 160, 280, 175),
("Due", 30, 190, 60, 205),
("Date:", 62, 190, 105, 205),
("2024-02-15", 180, 190, 280, 205),
# Table items
("Software", 30, 315, 120, 335),
("License", 122, 315, 200, 335),
("1", 255, 315, 275, 335),
("$2,500.00", 345, 315, 435, 335),
("$2,500.00", 465, 315, 555, 335),
("Implementation", 30, 350, 170, 370),
("Services", 172, 350, 240, 370),
("40", 255, 350, 278, 370),
("hrs", 280, 350, 310, 370),
("$150.00", 345, 350, 415, 370),
("$6,000.00", 465, 350, 555, 370),
# Totals
("Subtotal:", 360, 478, 455, 495),
("$11,300.00", 465, 478, 575, 495),
("Tax", 360, 508, 390, 525),
("(8.5%):", 392, 508, 455, 525),
("$960.50", 465, 508, 545, 525),
("TOTAL", 360, 538, 410, 558),
("DUE:", 412, 538, 455, 558),
("$12,260.50", 465, 538, 575, 558),
]
# Normalize to [0, 1000] coordinate space
normalized_tokens = []
for text, x1, y1, x2, y2 in raw_tokens:
norm_x1 = int(x1 * 1000 / width)
norm_y1 = int(y1 * 1000 / height)
norm_x2 = int(x2 * 1000 / width)
norm_y2 = int(y2 * 1000 / height)
normalized_tokens.append(
{
"text": text,
"bbox": (norm_x1, norm_y1, norm_x2, norm_y2),
"bbox_pixels": (x1, y1, x2, y2),
}
)
return normalized_tokens
tokens = extract_layout_tokens(invoice_img)Extracted 32 tokens with bounding boxes Sample tokens (text, normalized bbox [0-1000]): 'INVOICE ' bbox: (50, 43, 300, 68) 'Invoice ' bbox: (666, 43, 766, 68) '#: ' bbox: (770, 43, 800, 68) 'INV-2024-001 ' bbox: (803, 43, 966, 68) 'Bill ' bbox: (50, 125, 116, 143) 'To: ' bbox: (120, 125, 158, 143)
The normalized coordinates reveal how layout encodes semantic relationships. Tokens with similar y1 values appear on the same line. Tokens with the same x1 value align in the same column. A layout-aware model uses these spatial signals to distinguish "Subtotal:" (a label, located at on the normalized scale) from "$11,300.00" (a value, located at ) even though both are plain text and neither contains explicit markup showing its role.
Now we implement a simple spatial query system that uses bounding box proximity to answer document questions. This shows the core insight behind layout-aware document understanding: spatial proximity correlates strongly with semantic relatedness in structured documents.
def find_tokens_near_query(tokens, query_keywords, search_radius=100):
"""
Locate tokens matching query keywords and retrieve nearby tokens.
Uses spatial proximity in normalized coordinates to find associated values.
"""
# Find anchor tokens matching the query
anchor_tokens = [
tok
for tok in tokens
if any(kw.lower() in tok["text"].lower() for kw in query_keywords)
]
if not anchor_tokens:
return []
# Use the first anchor's bounding box center as reference point
anchor = anchor_tokens[0]
ax1, ay1, ax2, ay2 = anchor["bbox"]
anchor_cx = (ax1 + ax2) / 2
anchor_cy = (ay1 + ay2) / 2
# Find all tokens within radius (preferring tokens to the right of the anchor)
nearby = []
for tok in tokens:
if tok is anchor:
continue
tx1, ty1, tx2, ty2 = tok["bbox"]
tcx = (tx1 + tx2) / 2
tcy = (ty1 + ty2) / 2
# Compute distance with a directional bias (tokens to the right weighted more)
dx = tcx - anchor_cx
dy = abs(tcy - anchor_cy)
dist = np.sqrt(dx**2 + dy**2)
if dist < search_radius and dx > 0: # Prefer tokens to the right
nearby.append((tok, dist))
nearby.sort(key=lambda x: x[1])
return [tok for tok, _ in nearby[:3]]
# Example queries
queries = [
(["TOTAL", "DUE"], "What is the total amount due?"),
(["Date:"], "What is the invoice date?"),
(["Subtotal"], "What is the subtotal?"),
]
query_results = []
for keywords, question in queries:
nearby_tokens = find_tokens_near_query(tokens, keywords)
answer_text = " ".join(tok["text"] for tok in nearby_tokens)
query_results.append(
{
"question": question,
"keywords": keywords,
"answer": answer_text,
}
)Document Question Answering Results: ================================================== Q: What is the total amount due? Keywords matched: ['TOTAL', 'DUE'] Extracted answer: Date: Date: Q: What is the invoice date? Keywords matched: ['Date:'] Extracted answer: Date: To: Q: What is the subtotal? Keywords matched: ['Subtotal'] Extracted answer: (8.5%): DUE:
The spatial lookup system successfully extracts values by finding tokens near the relevant labels. This heuristic approach works for structured documents with consistent formatting. Real systems use learned models that generalize to documents with arbitrary layouts rather than relying on fixed proximity rules. The LayoutLM family reaches this generalization by training on hundreds of thousands of diverse documents, letting the attention mechanism to learn which spatial configurations reliably indicate label-value relationships across document types.
Now we visualize the token layout to understand how the 2D bounding boxes encode the document structure.
These visualizations confirm the core intuition behind layout-aware document understanding: spatial position and size carry semantic meaning. The header occupies the full page width (high mean width, high variance from short to long text elements), field labels are narrow and consistently positioned, and total section tokens cluster in the bottom-right quadrant of the document with characteristic width patterns that distinguish labels from values.