Scaling Up without Breaking the Bank › Horizontal Scaling: Running Multiple Instances

Michael BrenndoerferAugust 28, 20253 min read
AI Agent HandbookAll chapters
Part 15 of 15Performance and Cost Optimization - Doing More with Less
Chapter 44Scaling Up without Breaking the Bank › Horizontal Scaling: Running Multiple Instances
Reading section 3 of 13 · Horizontal Scaling: Running Multiple Instances · 3 minSection 3 of 13 · 3 min

Book contents

AI Agent Handbook15 parts · 44 chaptersView the whole book →
Active chapter 44 of 44

Section 3 of 13 · 3 min read

Scaling Up without Breaking the Bank › Horizontal Scaling: Running Multiple InstancesOne section · Sections in this chapter3 / 13 ↓
Sections in this chapter13

The most effective way to handle increased load is horizontal scaling. Instead of running one instance of your agent, you run multiple copies simultaneously. Each instance handles a portion of the incoming requests.

Think of it like a restaurant. One chef can serve maybe 20 customers per hour. But if you hire three chefs, you can serve 60 customers in the same time. Each chef works independently, handling different orders.

Here's how this works for our agent:

In[3]:
Code
## Using Claude Sonnet 4.5 for its superior agent capabilities
from anthropic import Anthropic
from concurrent.futures import ThreadPoolExecutor
import time

client = Anthropic(api_key=os.getenv("ANTHROPIC_API_KEY"))


def process_request(user_query):
    """Process a single user request"""
    start_time = time.time()

    response = client.messages.create(
        model="claude-sonnet-4-5",
        max_tokens=1024,
        messages=[{"role": "user", "content": user_query}],
    )

    elapsed = time.time() - start_time
    return {
        "query": user_query,
        "response": response.content[0].text,
        "time": elapsed,
    }


## Handle multiple requests concurrently
def handle_requests(queries, max_workers=5):
    """Process multiple queries using a pool of workers"""
    with ThreadPoolExecutor(max_workers=max_workers) as executor:
        results = list(executor.map(process_request, queries))
    return results


## Simulate multiple users
queries = [
    "What's the weather like today?",
    "Schedule a meeting for tomorrow at 2pm",
    "Summarize my emails from this morning",
    "Calculate 15% tip on $87.50",
    "Find flights to Boston next week",
]

results = handle_requests(queries, max_workers=3)

for result in results:
    print(f"Query: {result['query']}")
    print(f"Time: {result['time']:.2f}s\n")
Out[3]:
NameError: name 'os' is not defined
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[3], line 6
      3 from concurrent.futures import ThreadPoolExecutor
      4 import time
----> 6 client = Anthropic(api_key=os.getenv("ANTHROPIC_API_KEY"))
      9 def process_request(user_query):
     10     """Process a single user request"""

NameError: name 'os' is not defined

This example uses a thread pool to process multiple requests simultaneously. With max_workers=3, we can handle three requests at once instead of processing them one by one.

Let's see the difference:

Sequential Processing (one at a time):
Query 1: 2.1s
Query 2: 2.3s
Query 3: 1.9s
Query 4: 2.0s
Query 5: 2.2s
Total time: 10.5s

Concurrent Processing (3 workers):
Query 1: 2.1s
Query 2: 2.3s
Query 3: 1.9s
Query 4: 2.0s
Query 5: 2.2s
Total time: 4.3s

Notice that the individual query times don't change, but the total time drops dramatically. We're processing multiple requests in parallel rather than waiting for each to finish.

Choosing the Right Number of Workers

How many workers should you use? This depends on several factors:

API rate limits: Most AI providers limit how many requests you can make per minute. Claude Sonnet 4.5, for example, typically allows 50 requests per minute on standard plans. If you create too many workers, you'll hit rate limits and requests will fail.

Available resources: Each worker consumes memory and CPU. On a typical server, you might run 5-10 workers comfortably. On a powerful machine, you could run 20 or more.

Request patterns: If requests come in bursts (everyone uses the agent at 9am), you need more workers. If usage is steady throughout the day, fewer workers suffice.

A good starting point is to match your worker count to your expected concurrent users. If you typically have 5-10 people using the agent simultaneously, start with 10 workers. Monitor performance and adjust from there.