Scaling Up without Breaking the Bank › Glossary

Horizontal Scaling: Adding more instances of your application to handle increased load, rather than making a single instance more powerful. Like hiring more chefs instead of training one chef to cook faster.

Load Balancer: A system that distributes incoming requests across multiple server instances so no single instance gets overwhelmed while others sit idle.

Rate Limiting: Restricting how many requests a user or system can make within a time period, preventing overuse and controlling costs.

Caching: Storing the results of expensive operations so they can be reused without repeating the work. For agents, this means saving responses to common questions.

Cost Per Token: The price charged by AI providers for processing input tokens (the prompt) and generating output tokens (the response). Different models have different rates.

Concurrent Processing: Handling multiple requests simultaneously rather than one at a time, using techniques like thread pools or multiple server instances.

Monitoring Window: A time period over which you track metrics like request rate or response time. A 5-minute window shows recent trends without being overly sensitive to individual spikes.

Budget Threshold: A spending limit that triggers alerts or actions. You might set alerts at 75% of budget and hard stops at 100%.