Continuous Feedback and Improvement › Building a Feedback Loop
AI Agent HandbookAll chapters Continuous Feedback and Improvement › Building a Feedback LoopOne section · Sections in this chapter3 / 12 ↓
A feedback loop is simple in concept: use what you learn from the agent's performance to make it better. Here's how this looks in practice.
Capture What Goes Wrong
First, you need to know when things go wrong. There are several ways to collect this information:
User feedback: The most direct approach. When users interact with your assistant, give them a way to signal if something went wrong. This could be as simple as a thumbs up or thumbs down button, or a "Report an issue" link. Some developers include a quick feedback form: "Was this answer helpful? If not, what went wrong?"
Automatic failure detection: Your agent can monitor itself for certain red flags. Did a tool call fail? Did the agent say "I don't know" when it should have known? Did it take an unusually long time to respond? Log these events automatically.
Manual review: Periodically review a sample of conversations. You might pick random interactions, or focus on edge cases (very short queries, very long conversations, or sessions where the user asked follow-up questions repeatedly).
Here's a simple example of capturing feedback in code:
## Using Claude Sonnet 4.5 for its superior agent reasoning capabilities
from anthropic import Anthropic
import json
from datetime import datetime
client = Anthropic(api_key=os.getenv("ANTHROPIC_API_KEY"))
def log_interaction(user_query, agent_response, success, feedback=None):
"""Log each interaction for later review."""
log_entry = {
"timestamp": datetime.now().isoformat(),
"query": user_query,
"response": agent_response,
"success": success,
"feedback": feedback
}
# In practice, you'd write this to a database or file
with open("agent_feedback.jsonl", "a") as f:
f.write(json.dumps(log_entry) + "\n")
## Example usage
user_query = "What's the weather like for my trip next week?"
response = "The weather forecast for next week shows sunny skies with temperatures around 75°F."
## User indicates this was unhelpful
log_interaction(user_query, response, success=False, feedback="Wrong dates - didn't check my calendar")This logs every interaction along with whether it succeeded. Over time, you'll build a dataset of what works and what doesn't.
Analyze Patterns
Once you're collecting feedback, look for patterns. This is where the learning happens. You're not trying to fix every single mistake individually. Instead, you want to understand categories of problems.
Let's say after a week of use, you review your logs and notice:
- 15 queries where the agent misunderstood relative time phrases ("next week," "tomorrow," "in a few days")
- 8 queries where the agent used a tool but got an error and didn't recover gracefully
- 5 queries where the agent gave overly long answers when the user clearly wanted something brief
These patterns tell you where to focus your improvement efforts. The time phrases are the biggest issue, affecting the most users. That's your priority.
Make Targeted Improvements
Now you can improve your agent based on what you learned. For the time phrase issue, you might:
Refine the prompt: Add explicit guidance about handling relative time. For example: "When the user mentions relative time phrases like 'next week' or 'tomorrow,' always check the calendar tool first to determine the exact dates they mean."
Add examples: Include few-shot examples in your prompt showing how to handle these cases correctly.
Improve tool descriptions: Make sure your calendar tool description explicitly mentions it can resolve relative dates.
Here's how you might update the system prompt:
## Using Claude Sonnet 4.5 for agent tasks
from anthropic import Anthropic
client = Anthropic(api_key=os.getenv("ANTHROPIC_API_KEY"))
## Original prompt (had issues with relative time)
original_prompt = """You are a helpful personal assistant with access to tools.
Help the user with their requests using the available tools when needed."""
## Improved prompt based on feedback
improved_prompt = """You are a helpful personal assistant with access to tools.
When handling time-related requests:
- If the user mentions relative time ("next week", "tomorrow", "in a few days"), ALWAYS check the calendar tool first to determine exact dates
- Never assume relative time without checking the current date and the user's schedule
Example:
User: "What's the weather for my trip next week?"
Your thinking: I need to check the calendar to see when "next week" is and if there's a trip scheduled.
Action: Use calendar_tool to find trips in the next 7-14 days
Then: Use weather_tool with the specific dates found
Help the user with their requests using the available tools when needed."""
## Now when you make calls, use the improved prompt
message = client.messages.create(
model="claude-sonnet-4-5",
max_tokens=1024,
system=improved_prompt,
messages=[
{"role": "user", "content": "What's the weather like for my trip next week?"}
]
)The key is making changes that address categories of issues, not one-off fixes.
Test the Improvements
After making changes, you need to verify they actually help. This is where your test cases from Chapter 11.2 come back into play.
First, add new test cases based on the problems you found. For the relative time issue, you'd add tests like:
test_cases = [
{
"query": "What's the weather for my trip next week?",
"expected_behavior": "Should check calendar first, then check weather for specific dates found"
},
{
"query": "Remind me about my meeting tomorrow",
"expected_behavior": "Should identify tomorrow's date before searching calendar"
},
{
"query": "What do I have planned in a few days?",
"expected_behavior": "Should interpret 'a few days' as 2-4 days from now"
}
]Run these new tests to verify your changes fixed the problem. Also run your existing tests to make sure you didn't break anything that was working before. This is called regression testing, and it's essential. Sometimes a fix for one problem introduces a new problem elsewhere.
Deploy and Monitor
Once you're confident in your improvements, deploy the updated agent. But your work isn't done. Continue monitoring to see if the changes actually helped in production.
Check your feedback logs after the update. Did the number of relative-time errors decrease? Are users giving more positive feedback? You're looking for evidence that your changes made a real difference.
Sometimes you'll discover that your fix didn't fully solve the problem, or it introduced a new edge case. That's fine. You'll capture that feedback and loop back to the improvement step.