How often have you built a chatbot and spent days thinking through every possible workflow only to deploy it and see that users never use them? Our guess: too often. The bigger issue is that your chatbot analytics tool may have marked this case as successfully closed as it contained the conversation. In reality, the customer interaction was negative—but your tool didn't account for that.

Building and implementing a chatbot is not a one-time task. You need to regularly evaluate its key performance indicators (KPIs) to understand whether it’s impacting your bottom line. Simply put, you can’t improve its performance without measuring the right metrics. 

There are three broad categories of chatbot analytics you need to account for:

• Engagement metrics: You can measure how users interact with the chatbot and its interaction level. You can use the data to decide if your strategy and conversation flows are working.
• Conversion metrics: You can measure the number of conversions the chatbot achieves. The conversion goals can differ depending on the industry vertical and chatbot strategy. For example, software companies might measure demo sign-ups via chatbot conversations.
• Retention metrics: You can measure your chatbot's relevance and acceptance by your target audience. It can indicate the success or failure of your chatbot strategy.

Based on these categories, you can pinpoint specific metrics you need to track under each category and create a holistic chatbot strategy.

In this article, we'll discuss critical chatbot analytics you should track to improve your chatbots—and maximize their return on investment (ROI).

Which chatbot metrics should you track?

Before understanding which metrics to track, let's understand what the term 'chatbot analytics' means. 

Chatbot analytics or chatbot metrics refers to the real-time data your bot produces every time it interacts with users. The benefits include: 

• Identifying customer issues via frequently asked questions
• Determining how your website contributes to your lead generation process
• Understanding how customers currently engage with your brand

In a recent press release, Uma Challa, Senior Director Analyst, Gartner Customer Service & Support, said, "Customer Support & Service leaders have a positive future outlook for chatbots, but struggle to identify actionable metrics, minimizing their ability to drive chatbot evolution and expansion, and limiting their ROI." So, how do you know which KPIs to monitor to analyze its performance and success?

Here's a list of critical metrics you need to look out for to test your chatbot's performance.

User Metrics

When we talk about users, we mean potential or existing customers. To understand how your chatbot is anchoring them to your website, here are a few metrics you can track:

Total number of users

The total number of users refers to the number of users interacting with the chatbot over its entire lifetime. You can use this information to determine how many users interact with your chatbots and pinpoint granular metrics like:

• New users 
• Active users
• Engaged users

By measuring this KPI, you can determine whether or not your chatbot is enticing website/ app visitors. That's how you know your chatbot's design, call-to-action (CTA), and initial conversation prompt work well.

New users

New users refer to the number of unique users interacting with your chatbot within a specific period. This metric can gauge the success of external promotion efforts like paid advertising and organic social campaigns. You can also use this data to identify your current conversion rate by quantifying the number of leads generated via these campaigns.

Engaged users

Engaged users refer to active users who have conversations with your chatbot daily or weekly. This KPI can be used to understand how many users find your bot engaging—and prefer interacting with it to answer their queries. 

There are two key benefits of a high number of engaged users:

• It could lead to potential or recurring business over time
• It indicates the successful implementation of your chatbot

Chat volume

Chat volume refers to the number of successful interactions between users and the chatbot. By measuring this number—you can get more clarity on its usefulness, and the success of its conversation flows. The higher the chat volume, the better the capabilities of your chatbot.

It helps you understand whether users find it easy to converse with the chatbot and how frequently a specific user is conversing with it. The latter overlaps as an engagement metric too.

Goal completion rate (GCR)

Goal completion rate refers to the rate at which the chatbot achieves specific goals the company sets. The key distinction between GCR and conversion metrics is that the former focuses on goals for various purposes—not just conversion. E.g., if the goal is to resolve 50 user requests in a day, and it resolves 40 of them, the bot achieves an 80% GCR.

Engagement Metrics

You can use these metrics to identify how your users interact with the chatbot and the common queries they have in mind. Here are a few metrics to measure engagement:

Conversation length

Conversation length refers to the total length of the bot’s and user’s interaction. A longer conversation can mean that the user is engaging with the chatbot, it doesn’t always mean that they’re a positively engaged customer. 

If the user closes the conversation due to lack of options, it’s an unsuccessful interaction. Instead, you can measure the average conversation length and determine whether it works for specific use cases.

Average conversation length = Total duration of conversations in a specific timeframe (seconds) / Total number of conversations in the same timeframe

Bounce rate

Bounce rate refers to the number of users who visit the page but don't interact with the chatbot. It also detects the number of users who visit a single page and leave without interacting with other pages on the website. You can use this metric to determine if the website's design and content are interfering with the chatbot or if the chatbot fails to satisfy user intent.

Flow completion rate (FCR)

Flow competition rates (FCR) refer to the number of conversations that reach the endpoint of the conversation flow. It's useful when you want to understand whether the flow engages the user, and helps them find answers to their questions. 

Flow completion rate = Total number of completed flows / Total number of initialized flows

Customer Satisfaction Metrics

From a business perspective, if these conversational analytics fall short in these aspects, it could impact the overall customer experience. Here are a few metrics to measure customer satisfaction:

Retention rate

Retention rate refers to the percentage of users that return and engage with the chatbot within a specific timeframe. This KPI can be used to identify if customers are truly becoming part of your funnel (and eventually converting)—but it’s context-dependent.

E.g., if the customer returns the next day with the same issue they reported the previous day, the chatbot was ineffective in helping them.

Fallback rate (FBR)

Fallback rate (FBR) refers to the number of conversations that fail to understand the user's query. This metric is handy for businesses that use rule-based chatbots. A high FBR could indicate that the flow is not satisfactory and yields low user satisfaction. If a user chooses to speak to a support agent too quickly, it points toward a potential issue with the conversation flow.

Customer satisfaction score

The users provide customer satisfaction scores based on their experience with the chatbot. It's a straightforward way to gauge whether the quality of customer experience was optimal. Typically, the bot does this by offering a yes/ no question or a sliding scale (1-5 or 1-10). These options allow the company to gather feedback with minimal resistance as users can provide their inputs via a chat box.

Human takeover rate or escalation rate

Human takeover rate (HTR) or escalation rate refers to the number of times a customer executive had to take over the conversation and address the user’s questions. Usually, a high HTR indicates that the conversation flow is not good enough and needs to be improved. 

Although, there's a caveat. In specific scenarios, the next step would naturally involve engaging human agents—so it would be wrong for the bot to be deemed a failure.

Leads captured

This metric refers to the number of users who gave their personal information, like an email address or phone number, during the conversation. Most businesses use chatbots for lead generation, and it's an excellent way to capture high-quality leads as it can use qualifying questions to vet the lead and capture their data.

It's important to note that not all chatbots need to offer this option, but it's one of the most direct ways you can use a chatbot for lead generation.

Dimension Labs: Measure key chatbot analytics with ease

Now that we know what bot metrics you need to track to analyze its performance levels let's look at a chatbot analytics tool that can do all this for you.

Dimension Labs helps you easily understand all of the metrics that define success for your bot initiatives. It benchmarks and monitors the success of your bots and provides easy-to-interpret reports. 

You can monitor changes over time and receive alerts via our customizable dashboards to understand where the chatbot is improving or underperforming. It allows you to prioritize improvement projects with reports identifying underperformance factors quickly. Plus, you can integrate all your conversational data into one place and create a chatbot analytics  dashboard where learnings can be shared across your entire organization.

Track the right chatbot metrics and maximize its ROI

Listening to your users and using what they say can be a powerful way to improve your chatbot performance. Once you have launched your chatbot, the real work begins. It's essential to create a feedback loop between what workflows your bot currently handles, use conversational analytics to identify problems and make iterative changes to improve your KPIs. 

The more relevant data you have, the better you can optimize your chatbot for conversational success. Fortunately, chatbot analytics tools like Dimension Labs help you get these insights without having to do the heavy lifting.

If you are looking for a better way to measure, analyze, and improve your chatbot's performance, sign up for a demo today.