Refining the Agent Using Observability › Turning Observations into Improvements

Michael BrenndoerferAugust 10, 20253 min read
AI Agent HandbookAll chapters
Part 12 of 15Observability and Debugging - Peering Inside the Agent
Chapter 35Refining the Agent Using Observability › Turning Observations into Improvements
Reading section 3 of 11 · Turning Observations into Improvements · 3 minSection 3 of 11 · 3 min

Book contents

AI Agent Handbook15 parts · 44 chaptersView the whole book →
Active chapter 35 of 44

Section 3 of 11 · 3 min read

Refining the Agent Using Observability › Turning Observations into ImprovementsOne section · Sections in this chapter3 / 11 ↓
Sections in this chapter11

Once you've identified patterns, you can make targeted improvements. Let's walk through some examples.

Example 1: Addressing Knowledge Gaps

Suppose your logs show the agent frequently says "I don't know" when asked about recent events. The pattern is clear: it lacks access to current information.

The fix: Add a news search tool or update its knowledge base more frequently.

In[3]:
Code
## Using Claude Sonnet 4.5 for its superior tool-use capabilities
from anthropic import Anthropic

client = Anthropic(api_key=os.getenv("ANTHROPIC_API_KEY"))

## Define a news search tool
tools = [
    {
        "name": "search_news",
        "description": "Search for recent news articles about a topic. Use this when the user asks about current events or recent developments.",
        "input_schema": {
            "type": "object",
            "properties": {
                "query": {
                    "type": "string",
                    "description": "The search query for news articles",
                }
            },
            "required": ["query"],
        },
    }
]


def search_news(query):
    """Simulate a news search (in practice, call a real news API)."""
    # This would call a real news API
    return (
        f"Recent articles about {query}: [Article 1], [Article 2], [Article 3]"
    )


## Now when the agent encounters a question about recent events,
## it has a tool to help instead of saying "I don't know"
response = client.messages.create(
    model="claude-sonnet-4-5",
    max_tokens=1024,
    tools=tools,
    messages=[
        {
            "role": "user",
            "content": "What happened in the tech industry this week?",
        }
    ],
)

print(response.content)
Out[3]:
NameError: name 'os' is not defined
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[3], line 4
      1 ## Using Claude Sonnet 4.5 for its superior tool-use capabilities
      2 from anthropic import Anthropic
----> 4 client = Anthropic(api_key=os.getenv("ANTHROPIC_API_KEY"))
      6 ## Define a news search tool
      7 tools = [
      8     {
      9         "name": "search_news",
   (...)     21     }
     22 ]

NameError: name 'os' is not defined

After adding this tool, you'd monitor the logs again. You should see fewer "I don't know" responses to news-related questions and more successful tool calls to search_news.

Example 2: Optimizing Tool Usage

Your logs might show the agent calling the web search tool for information it already has in its conversation history. This wastes time and costs money.

The fix: Adjust the prompt to encourage the agent to check its memory first.

In[4]:
Code
## Improved system prompt that emphasizes checking memory first
system_prompt = """You are a helpful personal assistant.

Before using any tools, always check if you already have the information you need:

1. Review the conversation history
2. Check if the user has already provided this information
3. Only use tools if you genuinely need new information

This saves time and provides faster responses."""

## The agent will now be more thoughtful about when to use tools

Example 3: Streamlining Reasoning

If your logs show the agent's reasoning chains are consistently long and winding, you can guide it toward more efficient thinking.

The fix: Add examples of concise reasoning to the prompt.

In[6]:
Code
system_prompt = """You are a helpful personal assistant.

When reasoning through problems, be concise and direct:

Good example:
User asks: "What's 15% of 80?"
Reasoning: Need to calculate 15% of 80. That's $0.15 \times 80 = 12$.
Answer: 12

Avoid overthinking simple problems. Break down complex problems, but keep each step clear and necessary."""