Monitoring and Reliability › Handling Load: When Many Users Arrive
AI Agent HandbookAll chapters Monitoring and Reliability › Handling Load: When Many Users ArriveOne section · Sections in this chapter8 / 14 ↓
So far we've focused on keeping a single agent instance healthy. But what happens when your agent becomes popular and many people want to use it at once?
A single agent instance can only handle one request at a time (or a small number if you use async programming). If 100 people try to use the agent simultaneously, most of them will wait. If 1000 people try, the agent might crash from the load.
This is where scaling comes in. Scaling means increasing your agent's capacity to handle more requests.
Vertical Scaling
The simplest approach is vertical scaling: giving your agent more resources. If it's running on a server with 1GB of RAM and 1 CPU core, you could upgrade to 4GB of RAM and 4 CPU cores. This lets the agent handle more requests before running out of resources.
Most cloud platforms make vertical scaling easy. You change a setting to use a larger instance size, restart your agent, and it now has more capacity.
Vertical scaling has limits, though. Eventually you hit the maximum size available, and it gets expensive to keep upgrading to bigger machines.
Horizontal Scaling
A more flexible approach is horizontal scaling: running multiple copies of your agent. Instead of one powerful instance, you run several smaller instances. A load balancer distributes incoming requests across all the instances.
For example, if one instance can handle 10 requests per second, running 5 instances gives you capacity for 50 requests per second. If you need more capacity, you add more instances.
Here's the conceptual setup:
User Requests → Load Balancer → Agent Instance 1
→ Agent Instance 2
→ Agent Instance 3
→ Agent Instance 4Each agent instance is identical. They all run the same code. The load balancer just picks which one should handle each request, spreading the work evenly.
Most cloud platforms and deployment services support horizontal scaling. You configure how many instances to run, and the platform handles the load balancing automatically.
Stateless Design
For horizontal scaling to work well, your agent should be stateless. This means each request should be self-contained, not depending on information stored in the agent's memory from previous requests.
If your agent stores conversation history in memory (a Python dictionary or list), that history only exists in one instance. If the next request from the same user goes to a different instance, that instance won't have the history.
The solution is to store state externally, in a database or cache that all instances can access:
## Instead of storing in memory
conversation_history = {}
## Store in a shared database
def get_conversation_history(user_id):
return database.get(f"history:{user_id}")
def save_conversation_history(user_id, history):
database.set(f"history:{user_id}", history)Now any instance can retrieve the conversation history for any user. The agent instances themselves are stateless, they just read and write to the shared database.
This is a significant architectural change, but it's necessary for scaling beyond a single instance. The good news is that if you design for statelessness from the start, scaling later becomes straightforward.