AI agent evaluation tests whether an AI agent gives correct answers, follows business rules, and completes tasks reliably, before launch and after every change. LLM evaluation is the narrower part of it that scores a language model's individual outputs, such as the accuracy or tone of one answer.
For a customer service agent, the unit of evaluation is usually a whole conversation and its result. Was the refund issued, to the right customer, for the right amount, with the required disclosure?
What an AI agent evaluation measures
Answer accuracy. Answers match the knowledge base and current policy, with no invented facts (see AI accuracy and AI hallucination).
Task completion. The agent called the right tools with the right inputs, and the business systems ended in the right state.
Policy compliance. Identity checks, disclosures, and escalation rules were followed in every conversation where they apply.
Conversation quality. Tone, clarity, the number of turns, and whether a handover to a person carried the context with it.
Consistency. The agent reaches the same result when the same request is phrased differently or run again.
LLM evaluation and agent evaluation
LLM evaluation scores single outputs. Common methods are comparison with reference answers, rubrics scored by people, and LLM-as-a-judge, where a second model grades the answer. In a 2023 study, Judging LLM-as-a-Judge, strong model judges reached "over 80% agreement" with human preferences, about the level of agreement between humans. The same study documented position, verbosity, and self-enhancement biases in model judges.
Agent evaluation adds the actions. τ-bench, a 2024 research benchmark, is set in simulated retail and airline customer service. An agent with API tools and policy rules talks to a simulated customer. The conversation is scored by comparing the final database state with the expected one. Its pass^k metric is the chance that all k repeated trials of a task succeed, which measures consistency as well as success.
Offline and online evaluation
Offline, before release. A regression set of test conversations covers each request type and each regulated flow. Simulated customers run the set many times against the new version, and a failing result can hold the release back.
Online, in production. Live conversations are scored against the same criteria (see AI quality monitoring). Automated scoring can cover every conversation, and reviewers then read the ones it flags.
Diagnosis. When a conversation fails, a step-by-step record of what the agent saw, decided, and called shows where it went wrong (see Traces). The failed case then joins the regression set, so the same mistake is caught next time.
It helps to agree the criteria before a pilot starts, so that an old system and a new one are compared on the same terms.
Frequently asked questions
What is the difference between LLM evaluation and AI agent evaluation?
LLM evaluation scores a model's outputs one at a time, for accuracy, relevance, or tone. AI agent evaluation scores whole conversations, including the tools the agent called and the result in business systems. Agent evaluation usually includes LLM evaluation as one of its layers.
What is LLM-as-a-judge?
It is a method in which one language model grades another model's output against a rubric. It scales to thousands of conversations. Research has found biases, such as a preference for longer answers, so teams often calibrate the judge against a sample graded by people.
How many test conversations does an evaluation need?
There is no fixed number. A useful set covers every request type, every regulated flow, and the edge cases found in past incidents, with several phrasings of each. Running each test more than once shows whether the agent is consistent.
How often should an AI agent be evaluated?
Before every change reaches customers, including changes to the model, the prompts, the knowledge base, or an integration. In production, scoring runs continuously, and the team that owns the agent often reviews the results each week (see AgentOps).
How we approach this at Zowie
Zowie builds AI agents for voice, chat, and email, with over 100 million conversations a year across our clients. In each conversation, the language model speaks. A separate, deterministic Decision Engine returns every amount, rate, offer, disclosure, and escalation from your rules, with an audit log. You are welcome to book a demo.
Explore: AI quality monitoring, Traces, Supervisor, AI accuracy, AI hallucination, AgentOps