Answer
How to evaluate AI agents
Evaluating an AI agent means scoring its real output against explicit criteria, with evidence, at full volume. The practical method is: define what correct looks like, encode it as a scorecard, calibrate it against items humans already reviewed, then evaluate all traffic and group the failures.
The method, step by step
- 01
1. Write down failure modes first
List the ways your agent actually fails today — wrong answer, missing escalation, bad tool call, unsupported claim, abandoned task.
- 02
2. Turn each failure mode into a criterion
Give it a scale, a weight and a clear pass condition. Mark regulatory or safety breaches as hard critical failures.
- 03
3. Calibrate on reviewed items
Run 100–200 items your team already scored. Compare overall agreement, critical-failure recall and false-positive rate.
- 04
4. Refine the criteria, not the verdicts
Where the evaluation and the reviewer disagree, inspect the cited evidence and tighten the wording of the criterion.
- 05
5. Evaluate everything
Move from a sample to full coverage so intermittent failures stop hiding in the gaps.
- 06
6. Aggregate into a failure taxonomy
Group recurring failures, size them by volume and impact, and route them to prompt, tool, retrieval or product fixes.
Metrics worth tracking
Critical-failure rate
Share of interactions that breach a hard rule — the number that governs risk.
Criterion-level scores
Where quality is lost, rather than a single opaque overall number.
Task completion
Whether users reached resolution without escalation or repeat contact.
Agreement with human review
How closely automated scores track your reviewers after calibration.
Common mistakes
Scoring without evidence
A number you cannot trace cannot be defended to the team it judges.
One overall score only
Aggregates hide which behaviour actually regressed.
Evaluating only happy paths
Failures cluster in edge cases, handoffs and tool errors.
Never recalibrating
Agents, prompts and policies change; scorecards need versioning.