Setting Goals and Success Criteria › Types of Success Criteria

Different capabilities call for different kinds of success criteria. Let's explore the main categories.

Correctness measures whether the agent produces the right answer or performs the right action. This is your most fundamental criterion.

For our assistant:

• Factual questions: Does the agent provide accurate information? If asked "What's the capital of France?", does it say "Paris"?
• Calculations: Does the agent compute the correct result? If asked "What's 15% of $240?", does it return "$36"?
• Tool use: Does the agent call the right tool with the right parameters? If asked to "check the weather," does it invoke the weather API with the correct location?
• Task completion: Does the agent accomplish what the user requested? If asked to "schedule a meeting," does the event appear on the calendar?

For our personal assistant, we might set a correctness goal: "The agent should provide correct answers or complete tasks successfully in at least 90% of interactions."

Reliability measures consistency. An agent that works sometimes isn't good enough.

Consider these reliability dimensions:

• Consistency: Does the agent give the same answer to the same question asked twice?
• Robustness: Does the agent handle variations in phrasing? If a user says "What's tomorrow's weather?" versus "Tell me the weather forecast for tomorrow," does it work both times?
• Error handling: When something goes wrong (a tool fails, information is missing), does the agent handle it gracefully?

For our assistant, a reliability goal might be: "The agent should handle at least 95% of paraphrased queries correctly" or "The agent should gracefully handle tool failures without crashing or giving confusing responses."

Even correct answers lose value if they take too long. Efficiency criteria measure speed and resource usage.

For our assistant:

• Response time: How quickly does the agent respond? For a simple question, 2 seconds might be acceptable, but 30 seconds is not.
• Tool usage: Does the agent make unnecessary tool calls? If it can answer from memory, it shouldn't search the web.
• Cost: How much does each interaction cost in API calls? If you're using a paid model, efficiency directly affects your budget.

An efficiency goal might be: "90% of simple queries should receive responses within 3 seconds" or "The agent should complete multi-step tasks using no more than 5 tool calls on average."

Your agent should refuse inappropriate requests and respect boundaries.

Safety criteria for our assistant:

• Refusal handling: Does the agent politely decline requests it shouldn't fulfill?
• Privacy: Does the agent protect sensitive information and not leak it in responses?
• Permissions: Does the agent respect access controls? It shouldn't read files or access data it's not authorized to see.

A safety goal: "The agent should refuse 100% of requests that violate safety policies" or "The agent should never expose API keys or passwords in its responses."

Users judge an interaction by how it feels as well as whether the answer is correct.

• Clarity: Are the agent's responses clear and helpful? Or confusing and verbose?
• Tone: Is the agent appropriately professional, friendly, or conversational?
• Confirmation: For destructive or important actions, does the agent ask for confirmation before proceeding?

A UX goal: "95% of users should rate the agent's responses as 'clear and helpful' on a post-interaction survey."