Speeding Up the Agent › Caching Responses
Michael BrenndoerferAugust 24, 20252 min read
AI Agent HandbookAll chapters Part 15 of 15Performance and Cost Optimization - Doing More with Less
Speeding Up the Agent › Caching ResponsesOne section · Sections in this chapter6 / 15 ↓
Sections in this chapter15
If users frequently ask the same questions, why recompute the answer every time? Cache the response and serve it instantly on subsequent requests.
Example: Simple Response Cache (Gemini 2.5 Flash)
In[6]:
Codefrom google import genai
import hashlib
import time
client = genai.Client(api_key=os.getenv("GEMINI_API_KEY"))
class CachedAgent:
"""Agent with response caching for repeated queries."""
def __init__(self):
self.cache = {}
def _hash_query(self, query):
"""Create a cache key from the query."""
return hashlib.md5(query.encode()).hexdigest()
def respond(self, query):
"""Get response, using cache if available."""
cache_key = self._hash_query(query)
# Check cache first
if cache_key in self.cache:
print("Cache hit! Instant response.")
return self.cache[cache_key]
# Cache miss: call the model
print("Cache miss. Calling model...")
start = time.time()
# Using Gemini 2.5 Flash for fast responses
response = client.models.generate_content(
model="gemini-2.5-flash",
contents=query
)
elapsed = time.time() - start
result = response.text
self.cache[cache_key] = result
print(f"Response time: {elapsed:.2f}s")
return result
## Test the cache
agent = CachedAgent()
## First call: cache miss
print(agent.respond("What is machine learning?"))
print()
## Second call: cache hit (instant)
print(agent.respond("What is machine learning?"))Out[6]:
ConsoleCache miss. Calling model...
Response time: 14.16s
Machine learning (ML) is a **subset of artificial intelligence (AI)** that enables systems to **learn from data, identify patterns, and make decisions or predictions** with minimal human intervention, rather than being explicitly programmed for every task.
Think of it like teaching a child:
* You don't give them a detailed rulebook for every situation.
* Instead, you show them many examples (data).
* They learn to recognize patterns and make their own judgments based on those examples.
* Over time, with more experience and feedback, they get better at it.
Machine learning works in a very similar way, but for computers.
---
Here's a breakdown of the key concepts:
1. **Learning from Data:**
* Instead of being explicitly coded with a set of if-then rules, ML algorithms are fed large amounts of data (training data).
* This data contains examples of the task the machine needs to learn, often including both inputs and desired outputs.
2. **Pattern Recognition:**
* The algorithms analyze this data to find statistical relationships, correlations, and hidden patterns.
* They "learn" a model that represents these patterns.
3. **Making Predictions or Decisions:**
* Once trained, the ML model can then be used on new, unseen data.
* It applies the patterns it learned from the training data to make predictions, classify new inputs, or make decisions.
4. **Iterative Improvement:**
* Machine learning models can continuously improve their performance over time as they are exposed to more data and feedback.
---
### How it Generally Works (Simplified):
1. **Data Collection:** Gather relevant data (e.g., images, text, numbers, sensor readings).
2. **Feature Engineering:** Select and transform the most important characteristics (features) from the data that the model will learn from.
3. **Algorithm Selection:** Choose a suitable machine learning algorithm (e.g., linear regression, decision trees, neural networks).
4. **Training:** Feed the processed data to the algorithm. The algorithm adjusts its internal parameters to minimize errors and learn the underlying patterns. The output of this phase is a "trained model."
5. **Evaluation:** Test the trained model on new, unseen data to see how well it performs and generalize.
6. **Deployment:** Once the model is satisfactory, it can be put into production to make real-time predictions or decisions.
---
### Main Types of Machine Learning:
1. **Supervised Learning:**
* **Concept:** Learning from labeled data, where both input and the correct output are provided. Like learning with a teacher.
* **Tasks:**
* **Classification:** Predicting a category (e.g., spam or not spam, cat or dog).
* **Regression:** Predicting a continuous value (e.g., house prices, temperature).
* **Examples:** Spam detection, image recognition, medical diagnosis.
2. **Unsupervised Learning:**
* **Concept:** Learning from unlabeled data, finding hidden structures or patterns without explicit guidance. Like exploring on your own.
* **Tasks:**
* **Clustering:** Grouping similar data points together (e.g., customer segmentation).
* **Dimensionality Reduction:** Simplifying data while retaining important information.
* **Examples:** Recommender systems, anomaly detection, topic modeling.
3. **Reinforcement Learning:**
* **Concept:** An agent learns to make a sequence of decisions in an environment by performing actions and receiving rewards or penalties. Like learning through trial and error.
* **Tasks:** Finding an optimal strategy to achieve a goal.
* **Examples:** Game playing (AlphaGo), robotics, self-driving cars (parts of it).
---
### Why is Machine Learning Important?
* **Automation:** Automates tasks that are complex or impossible to program manually.
* **Scalability:** Can process and learn from vast amounts of data that humans cannot.
* **Discovery:** Uncovers insights and patterns that might be hidden within data.
* **Adaptability:** Models can adapt and improve over time with new data.
* **Personalization:** Powers customized experiences (e.g., recommendation engines).
---
### Common Applications:
* **Recommendation Systems:** (Netflix, Amazon, YouTube)
* **Spam Filters:** (Email services)
* **Fraud Detection:** (Banks, credit card companies)
* **Facial Recognition:** (Phone unlocks, security systems)
* **Speech Recognition:** (Siri, Alexa, Google Assistant)
* **Medical Diagnosis:** (Identifying diseases from scans)
* **Natural Language Processing:** (Translation, sentiment analysis)
* **Self-Driving Cars:** (Object detection, path planning)
In essence, machine learning is about empowering computers to **learn from experience** (data) and make intelligent decisions, much like humans do, but at an unprecedented scale and speed.
Cache hit! Instant response.
Machine learning (ML) is a **subset of artificial intelligence (AI)** that enables systems to **learn from data, identify patterns, and make decisions or predictions** with minimal human intervention, rather than being explicitly programmed for every task.
Think of it like teaching a child:
* You don't give them a detailed rulebook for every situation.
* Instead, you show them many examples (data).
* They learn to recognize patterns and make their own judgments based on those examples.
* Over time, with more experience and feedback, they get better at it.
Machine learning works in a very similar way, but for computers.
---
Here's a breakdown of the key concepts:
1. **Learning from Data:**
* Instead of being explicitly coded with a set of if-then rules, ML algorithms are fed large amounts of data (training data).
* This data contains examples of the task the machine needs to learn, often including both inputs and desired outputs.
2. **Pattern Recognition:**
* The algorithms analyze this data to find statistical relationships, correlations, and hidden patterns.
* They "learn" a model that represents these patterns.
3. **Making Predictions or Decisions:**
* Once trained, the ML model can then be used on new, unseen data.
* It applies the patterns it learned from the training data to make predictions, classify new inputs, or make decisions.
4. **Iterative Improvement:**
* Machine learning models can continuously improve their performance over time as they are exposed to more data and feedback.
---
### How it Generally Works (Simplified):
1. **Data Collection:** Gather relevant data (e.g., images, text, numbers, sensor readings).
2. **Feature Engineering:** Select and transform the most important characteristics (features) from the data that the model will learn from.
3. **Algorithm Selection:** Choose a suitable machine learning algorithm (e.g., linear regression, decision trees, neural networks).
4. **Training:** Feed the processed data to the algorithm. The algorithm adjusts its internal parameters to minimize errors and learn the underlying patterns. The output of this phase is a "trained model."
5. **Evaluation:** Test the trained model on new, unseen data to see how well it performs and generalize.
6. **Deployment:** Once the model is satisfactory, it can be put into production to make real-time predictions or decisions.
---
### Main Types of Machine Learning:
1. **Supervised Learning:**
* **Concept:** Learning from labeled data, where both input and the correct output are provided. Like learning with a teacher.
* **Tasks:**
* **Classification:** Predicting a category (e.g., spam or not spam, cat or dog).
* **Regression:** Predicting a continuous value (e.g., house prices, temperature).
* **Examples:** Spam detection, image recognition, medical diagnosis.
2. **Unsupervised Learning:**
* **Concept:** Learning from unlabeled data, finding hidden structures or patterns without explicit guidance. Like exploring on your own.
* **Tasks:**
* **Clustering:** Grouping similar data points together (e.g., customer segmentation).
* **Dimensionality Reduction:** Simplifying data while retaining important information.
* **Examples:** Recommender systems, anomaly detection, topic modeling.
3. **Reinforcement Learning:**
* **Concept:** An agent learns to make a sequence of decisions in an environment by performing actions and receiving rewards or penalties. Like learning through trial and error.
* **Tasks:** Finding an optimal strategy to achieve a goal.
* **Examples:** Game playing (AlphaGo), robotics, self-driving cars (parts of it).
---
### Why is Machine Learning Important?
* **Automation:** Automates tasks that are complex or impossible to program manually.
* **Scalability:** Can process and learn from vast amounts of data that humans cannot.
* **Discovery:** Uncovers insights and patterns that might be hidden within data.
* **Adaptability:** Models can adapt and improve over time with new data.
* **Personalization:** Powers customized experiences (e.g., recommendation engines).
---
### Common Applications:
* **Recommendation Systems:** (Netflix, Amazon, YouTube)
* **Spam Filters:** (Email services)
* **Fraud Detection:** (Banks, credit card companies)
* **Facial Recognition:** (Phone unlocks, security systems)
* **Speech Recognition:** (Siri, Alexa, Google Assistant)
* **Medical Diagnosis:** (Identifying diseases from scans)
* **Natural Language Processing:** (Translation, sentiment analysis)
* **Self-Driving Cars:** (Object detection, path planning)
In essence, machine learning is about empowering computers to **learn from experience** (data) and make intelligent decisions, much like humans do, but at an unprecedented scale and speed.The first call takes the normal time (maybe 1.5 seconds). The second call is instant, returning in milliseconds. For a production system, you'd use a more sophisticated cache with expiration times and size limits, but this shows the basic idea.