Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Testing the Agent with Examples › Tracking Performance Over Time

Michael BrenndoerferAugust 2, 20253 min read

AI Agent HandbookAll chapters

Part 11 of 15Evaluation - Measuring Your Agent's Performance

Chapter 31Testing the Agent with Examples › Tracking Performance Over Time

Reading section 5 of 11 · Tracking Performance Over Time · 3 minSection 5 of 11 · 3 min

Section 5 of 11 · 3 min read

Testing the Agent with Examples › Tracking Performance Over TimeOne section · Sections in this chapter5 / 11 ↓

Sections in this chapter11

Testing once is useful, but testing regularly is powerful. By running the same test suite after each improvement to your agent, you can see whether your changes help or hurt performance.

Here's a simple way to track results over time:

In[8]:

Code

import json
from datetime import datetime


def save_test_run(results, version_name):
    """
    Save test results with a timestamp for historical tracking.
    """
    run_data = {
        "timestamp": datetime.now().isoformat(),
        "version": version_name,
        "results": results,
        "summary": {
            "total": len(results),
            "passed": sum(1 for r in results if r["passed"]),
            "pass_rate": sum(1 for r in results if r["passed"]) / len(results),
        },
    }

    filename = f"test_results_{version_name}_{datetime.now().strftime('%Y%m%d_%H%M%S')}.json"
    with open(filename, "w") as f:
        json.dump(run_data, f, indent=2)

    print(f"Results saved to {filename}")
    return filename

Now you can track how your agent improves. Maybe you started with a pass rate of 60%, then after adding better tool selection logic you're at 75%, and after refining prompts you hit 90%. Each test run gives you confidence that you're moving in the right direction.

You can even create a simple visualization:

In[9]:

Code

import json
import sys
from pathlib import Path
from tempfile import TemporaryDirectory

import matplotlib.pyplot as plt

for parent in [Path.cwd(), *Path.cwd().parents]:
    helper_dir = (
        parent / "apps/web/content/notebooks/1_machine-learning-from-scratch"
    )
    if (helper_dir / "mls_plot_style.py").exists():
        if str(helper_dir) not in sys.path:
            sys.path.insert(0, str(helper_dir))
        break
else:
    raise FileNotFoundError("Could not locate mls_plot_style.py")

from mls_plot_style import PALETTE, polish_axes, use_mls_style


def plot_progress(test_history_files, output_path="agent_progress.png"):
    """
    Plot the agent's test performance over time.
    """
    versions = []
    pass_rates = []

    for file in test_history_files:
        with open(file, "r") as f:
            data = json.load(f)
            versions.append(data["version"])
            pass_rates.append(data["summary"]["pass_rate"] * 100)

    use_mls_style()
    fig, ax = plt.subplots(figsize=(6.0, 4.0))
    ax.plot(
        versions,
        pass_rates,
        color=PALETTE["primary"],
        marker="o",
        linewidth=2.2,
        markersize=5,
    )
    ax.set_xlabel("Version")
    ax.set_ylabel("Pass Rate (%)")
    ax.set_title("Agent Test Performance Over Time")
    ax.set_ylim(0, 100)
    polish_axes(ax)
    if output_path is not None:
        fig.savefig(output_path)
        print(f"Progress chart saved to {output_path}")
    return fig, ax


with TemporaryDirectory() as temp_dir:
    example_files = []
    for version, pass_rate in [("v1.0", 0.60), ("v1.1", 0.75), ("v1.2", 0.90)]:
        history_path = Path(temp_dir) / f"{version}.json"
        history_path.write_text(
            json.dumps(
                {"version": version, "summary": {"pass_rate": pass_rate}}
            ),
            encoding="utf-8",
        )
        example_files.append(history_path)

    fig, ax = plot_progress(example_files, output_path=None)

plt.show()

Out[9]:

Visualization

Line chart showing agent test pass rate increasing from 60 percent in version 1.0 to 75 percent in version 1.1 and 90 percent in version 1.2. — Agent test pass rate rises from 60% to 90% across three iterations.

Seeing your agent's performance improve over time is motivating. Testing becomes a feedback loop for continuous improvement rather than a chore.