Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Speeding Up the Agent › Measuring the Impact

Michael BrenndoerferAugust 24, 20251 min read

AI Agent HandbookAll chapters

Part 15 of 15Performance and Cost Optimization - Doing More with Less

Chapter 42Speeding Up the Agent › Measuring the Impact

Reading section 11 of 15 · Measuring the Impact · 1 minSection 11 of 15 · 1 min

Section 11 of 15 · 1 min read

Speeding Up the Agent › Measuring the ImpactOne section · Sections in this chapter11 / 15 ↓

Sections in this chapter15

As you apply these optimizations, measure their impact. Here's a simple benchmarking approach:

In[11]:

Code

import time
import os
from anthropic import Anthropic

client = Anthropic(api_key=os.getenv("ANTHROPIC_API_KEY"))

def benchmark_agent(queries, model="claude-sonnet-4-5", max_tokens=1024):
    """Measure average response time for a set of queries."""
    times = []
    
    for query in queries:
        start = time.time()
        response = client.messages.create(
            model=model,
            max_tokens=max_tokens,
            messages=[{"role": "user", "content": query}]
        )
        elapsed = time.time() - start
        times.append(elapsed)
    
    avg_time = sum(times) / len(times)
    min_time = min(times)
    max_time = max(times)
    
    return {
        'average': avg_time,
        'min': min_time,
        'max': max_time,
        'total': sum(times)
    }

## Test queries
test_queries = [
    "What is 47 times 83?",
    "What's the capital of France?",
    "How many days until Christmas?",
    "Convert 10 miles to kilometers",
    "What's the weather like today?"
]

## Benchmark before optimization
print("Before optimization:")
results_before = benchmark_agent(test_queries, max_tokens=1024)
print(f"Average: {results_before['average']:.2f}s")
print(f"Total: {results_before['total']:.2f}s")

## Benchmark after optimization (concise responses)
print("\nAfter optimization (concise responses):")
results_after = benchmark_agent(test_queries, max_tokens=200)
print(f"Average: {results_after['average']:.2f}s")
print(f"Total: {results_after['total']:.2f}s")

## Calculate improvement
improvement = (1 - results_after['average'] / results_before['average']) * 100
print(f"\nSpeed improvement: {improvement:.1f}%")

Out[11]:

Console

Before optimization:

Average: 2.92s
Total: 14.58s

After optimization (concise responses):

Average: 3.36s
Total: 16.78s

Speed improvement: -15.1%

This gives you concrete numbers to evaluate your optimizations. You might find that limiting tokens saves 20% on response time, or that caching cuts average latency by 40%.