Measuring an AI agent: the numbers that matter

Once an AI agent is live, someone will ask the obvious question: is it working? It's a fair question, and "it seems to be helping" is not a good enough answer. You're paying for the agent, your team is changing how it works around it, and your customers are on the other end of what it does.

The good news is that measuring an agent doesn't need a complicated dashboard. It needs a clear starting point, a handful of measures that match the role you gave it, and a regular habit of looking at them with the people who work alongside it.

Start with a baseline, before the agent goes live

The most common mistake is trying to measure improvement without knowing where you started. If nobody recorded how long customers waited for a first reply last month, you can't say much about how long they wait now.

So before launch, spend a little time writing down how the work runs today. It doesn't need to be perfect. Your helpdesk, CRM or order system usually holds most of what you need, and a few weeks of history is enough to give you a fair picture.

• How much of this work arrives in a normal week, and how it varies on busy days.
• How long it takes to get a first response and a final resolution.
• Who handles it today, and roughly how much of their week it takes.
• Where things currently go wrong: missed messages, late replies, errors that need fixing later.

Speed: how quickly things get moving

For most customer-facing agents, the first number people look at is time to first response. It matters because customers notice it straight away, and because a fast, accurate first reply often prevents a second or third message.

Look at time to resolution as well. A fast first reply that leads to a long, confused thread is not a win. And look at the spread, not just the average. If most replies are instant but a few wait overnight because they were escalated and nobody picked them up, the average will hide the problem your customers actually feel.

Resolution without a person, and the quality behind it

The second measure is how much work the agent completes on its own, from start to finish, within the limits you set. An order-status question answered with live tracking, a return approved inside the policy window, a lead qualified and booked into a site visit.

This number is only useful alongside a quality check. An agent that closes lots of conversations but leaves customers unhappy, or gets details wrong, is creating work rather than saving it. A simple routine helps here:

• Each week, a team lead reviews a sample of conversations the agent handled alone.
• Each one is marked as correct, acceptable with small issues, or wrong.
• Anything wrong is traced back to its cause: a gap in the SOP, missing data or an unclear rule.
• Reopened tickets and repeat contacts on the same issue are tracked as a warning sign.

A number going up is not the goal. The goal is the right work being done well, with people stepping in where they should. Measure the agent's quality as carefully as its volume.

Exceptions: what the agent hands back

Every well-designed agent hands some work to people. Refunds above your approval limit, complaints that need a human touch, anything it isn't confident about. Tracking these exceptions tells you a lot.

Watch how many there are, what kinds they are and how quickly your team picks them up. If the same type of exception keeps appearing, it may be a sign the agent needs a clearer rule or better access to data. If exceptions sit waiting for hours, the bottleneck has moved to your team, and that's worth fixing too. Some exceptions should never go away. If your approval limits are doing their job, a steady flow of refunds waiting for sign-off is the system working as intended.

Hours returned to your team

The measure owners care about most is time. How much of your team's week used to go on this work, and how much goes on it now? This is harder to measure precisely, so be honest about it. A rough estimate based on your baseline and the agent's activity log is fine, as long as everyone understands it's an estimate.

Then ask the more useful question: what is that time being spent on now? If your support lead is finally working on the returns policy, or your sales team is spending more time on site visits instead of chasing enquiries, that's the real return. Hours that simply disappear into other admin are worth noticing as well.

Making measurement a habit

Numbers only help if someone looks at them. A short weekly review in the first couple of months works well: the agent's numbers against the baseline, a few sample conversations, the open exceptions and any changes to SOPs or rules. Once things settle, monthly is usually enough.

Agree on what "good" looks like before you start, and decide what you'll do if a measure moves the wrong way. That keeps the review practical. It also gives you a sensible basis for deciding when to widen the agent's role, and when to hold back.

If you're planning an agent and want to know what you should be measuring from day one, book a call with us. We'll look at the work you want to hand over and help you set a baseline before anything goes live.