Measuring Conversational AI for Customer Service: KPIs and Dashboards

It covers chatbots and voice agents that understand free text or speech and reply in natural language. Modern systems combine a large language model with a knowledge base and, often, actions such as checking an order or booking an appointment.

Common platforms include Zendesk AI agents, Intercom Fin, Salesforce Agentforce, Microsoft Copilot Studio, Google Customer Engagement Suite, HubSpot’s customer agent and many smaller tools for WhatsApp and website chat. Most include their own analytics, but those reports tend to show the vendor’s preferred numbers.

Pick a small set that covers effectiveness, quality, experience and cost. More metrics dilute attention.

Effectiveness

• Containment rate: share of conversations that end without a human handoff
• Resolution rate: share where the customer’s issue was actually solved, confirmed by survey or by no repeat contact
• Repeat contact rate: customers who come back on the same issue within a set window, such as 72 hours

Quality

• Answer accuracy on a reviewed sample, scored by a person
• Hallucination or policy-breach count: answers that invented facts or broke rules
• Handoff quality: did the human agent receive the context, or did the customer repeat everything?

Experience

• CSAT after bot conversations, compared with CSAT after human ones
• Abandonment rate: customers who leave mid-conversation
• Time to resolution

Cost

• Cost per resolved conversation, including platform fees and staff time for maintenance
• Human agent hours freed, and what those hours were used for

Containment counts any conversation that did not reach a human. A customer who gives up and closes the window counts as contained. So does one who calls the phone line instead.

Pair containment with repeat contact and CSAT. If containment rises while repeat contacts and calls rise too, the bot is pushing work elsewhere, not removing it.

Measure the old setup for at least a month before launch: volume by topic, handling time, CSAT and cost per contact. Without a baseline, every improvement claim is a guess.

• Export ticket and chat data with topic tags
• Calculate average handling time per topic
• Record CSAT by channel
• Note seasonal peaks that could distort comparisons

After launch, compare the same topics over a similar period. If you can, run the bot on part of the traffic first and keep the rest on the old flow; that gives a cleaner comparison.

One page, four blocks, updated weekly. Leaders should be able to read it in two minutes.

• Top block: conversations, resolution rate, CSAT, cost per resolved contact, each with a trend line
• Topic table: volume, resolution and CSAT per topic, sorted by volume
• Quality block: accuracy score from the weekly sample, policy breaches, handoff complaints
• Gaps list: top unanswered questions and topics with falling resolution

Data usually comes from the bot platform, the helpdesk and survey tool. Tools like Looker Studio, Power BI or the helpdesk’s own reporting can combine them. Keep definitions written on the dashboard itself, so “resolved” means the same thing every month.

Automated metrics miss tone and subtle errors. A weekly sample review catches them.

• Sample 30 to 100 conversations per week, spread across topics
• Score each for accuracy, tone and correct handoff
• Log any wrong answer and fix the knowledge source behind it
• Share two good and two bad examples with the support team

Take a hypothetical online electronics shop that handles about 6,000 chats a month. Before launch, its baseline shows order status questions make up a large share of volume, with an average handling time of four minutes and CSAT around the team’s usual level.

After the AI agent goes live on order status, the dashboard shows high containment on that topic. Repeat contacts on order status stay flat and CSAT holds steady, so the containment looks real. On returns, though, containment is high while repeat contacts double. Reviewing samples shows the agent quoting an old return window from an outdated help page.

The fix is not in the AI settings. The help page is updated, the agent’s answers change the same day, and repeat contacts fall back the next week. This is the pattern good measurement produces: a specific number points to a specific page, and the fix is cheap.

Leaders want to know three things: is it working, is it safe and what does it cost. Answer those in the first lines of any report, then show detail below.

• One sentence on resolution and CSAT trend
• One sentence on quality incidents and fixes
• One sentence on cost per resolved contact against the human baseline

Avoid reporting containment alone as success. It invites the wrong question later, when complaints rise in another channel.