What sits behind the 98.8% number
AI agent accuracy is easy to measure in a way that looks good. Here is how we count it and why public benchmarks did not help.
We say agent answer accuracy in real business cases is 98.8%. That number is easy to obtain dishonestly, so let us unpack what stands behind it.
Why benchmarks did not help us
Public question sets measure a model's knowledge, not an agent's work inside a business. A travel agent does not get geography wrong. It gets things wrong when a client writes “same dates but cheaper” and the agent cannot tell which dates they meant.
So we measure accuracy only on the client's live conversations, and from scratch for every rollout.
What counts as a mistake
We split mistakes into four classes and count them separately, because they cost very different amounts.
- Factual error: the agent quoted a price, a condition or a deadline that is not in the knowledge base. The most expensive class, with an acceptable share of zero.
- Lost context: the agent missed a reference to something said earlier and asked a question that was already answered.
- Wrong routing: the agent decided alone where it should have escalated, or the other way round.
- Wrong tone: the answer is formally correct and reads wrong in the situation. The most arguable class, scored by hand.
How we count
Every week we take a random sample of conversations and label it in two passes. First a separate model works through a checklist across the four classes. Then a person from the rollout team reviews every case the model flagged plus 10% of the cases it did not. The second part exists to catch mistakes in the labelling itself.
Accuracy is the share of conversations with no mistake of any class. Not the share of correct messages: one wrong answer breaks the whole conversation, and counting it as “one message out of forty” means lying to yourself.
The unit of measurement is the conversation, not the message. A client does not leave because of a message, they leave because of a conversation.
What we do with mistakes
Factual errors almost always mean a hole in the knowledge base rather than a model problem. The agent said something untrue because nobody gave it the truth. That is fixed with a document, not a prompt.
Lost context is fixed by narrowing the task. If an agent gets confused in long tour-selection threads, we move selection into a separate flow with explicit state instead of adding instructions.
Wrong routing is the most useful class for the business. Every case points at a place where ownership is described badly. Usually the review ends with a change to the human playbook rather than to the agent.
What the number does not mean
98.8% is the result on rollouts that went through setup and have run for months. In week one with a new client accuracy is lower, and we keep the agent in suggestion mode: it drafts the reply, a manager approves it. The agent goes autonomous topic by topic, not all at once.