A Step-by-Step Guide for Testing Your Chatbot in 2018

It’s pretty dang hard to get around the internet these days without hearing about chatbots and ‘the future of AI’. It’s a hot topic and something we’re pretty excited about.

Screen-Shot-2018-01-30-at-2.25.33-PM.png#asset:120

I’m pretty strict about what I subscribe to and here is my inbox filtering for AI (so not including emails using chatbot specifically).

We’ve been doing a lot of research, learning more about what customers need and expect from their online experience, and helping clients implement chatbots of their own. There are a number of great blog posts out there (like this one from Drift) that dive deeper into what chatbots are all about and provide data about what consumers expect.

In this post, we’ll briefly cover what opportunities a chatbot can help you take advantage of but the true intention is to give you a framework and template for testing a chatbot on your own to see if it works well for your customers as well as employees.

Tell me more about chatbots...

Chatbots are Artificial Intelligence (AI) programs that can process and respond to simple queries from your audience like an interactive FAQ. You’ve seen them across the internet, often in the bottom right corner of a web page. It looks similar to live chat but with programmed data on the other end. When a company is utilizing a chatbot, they’ve taken the time to teach the chatbot the answers to questions they expect a customer to ask. It works a bit like a flow chart:

1_eNYl6B0L39uGiJD-ltNfQw.png#asset:121

We like this breakdown from Jesús Martín.

Examples of companies running chatbots...

  • Duolingo uses chatbots inside their language teaching app and allows users to practice conversation by simulating text exchanges. [Learn more about the bot here]

  • Apartment Ocean is a chatbot built to help real estate agents qualify leads and learn more about potential customers. [Learn more about the bot here]

  • Pizza Hut allows people to order pizzas and reorder their favorites via a Facebook messenger chatbot. [Test out their bot here]

  • Casper, a mattress company, has a purely promotional bot that is active from 11pm to 5am to ‘keep you company when you just can’t fall asleep’. While it can chat on many topics from Stranger Things to Seinfeld, it also takes the opportunity to plug their mattresses from time to time. [Learn more about the bot here]

Why should I bite and try a chatbot?...

Do you ask customers to fill out a form?: Chatbots have helped create a shift toward something being coined as ‘conversational marketing’. You can use chatbots to replace long forms with more intuitive and natural conversations. You can set up a bot to ask those same qualifying leads your forms are searching for. Depending on the potential customer’s answers, the chatbot will send them through the right flow.

Do you want to help your sales agents save time and close more leads?: A chatbot can automatically qualify leads and get them to the right agents. As research from InsideSales.com and the Harvard Business Review shows, even if you wait just five minutes to respond after a lead first reaches out, there’s a 10x decrease in your odds of actually getting in touch with that lead. After 10 minutes, there’s a 400% decrease in your odds of qualifying that lead. By automating this crucial step, your chatbot can quickly disqualify leads and get the most promising ones quickly to your agents.

Do you want to improve your customer experience?: There’s never going to be a future where Artificial Intelligence totally takes over because humans and chatbots are good at different things. Leaning on our strengths and the strengths of chatbots can make for a power team. When a chatbot pilot program was initiated in a telco company, it could handle 82% of common queries in customer service. After 5 weeks of tweaking, analyzing, and optimizing by human agents, its success increased to 88%, according to Accenture.

In the 2018 State of Chatbots study from Drift and friends, they surveyed over 1,000 internet users in the United States and made sure to match their audience to represent the U.S. adult online population. Here’s what the group had to say about their current online experiences:

NqMxIFKJ1OWPuY-JVSxnN-7LVbZNcidXM7uaTzLd1CN8aQCTYuYvIYoETQZnV6ByYYMRXgoGzxqLuXwdH3IifFAjYLNW4-T7he0RXkLyiiMrBdKNFrgJ-Ga-4YJZfBvjU8Xor3Dk

In addition to helping people get quicker answers, you can use the chatbot data to make changes to your website and try to eliminate the root cause of the most frequently asked questions.

Do you have customers spanning multiple generations?: One preconceived notion I had about chatbots is that they were better suited for companies with younger target audiences. I was excited to find data saying quite the opposite. In that same 2018 State of Chatbots Study, Drift found that Baby Boomers (age 55+) were 24% more likely than Millennials (age 18-34) to expect benefits from chatbots in five of the nine following categories:

24gXreVvsPNYhyDepiIP_35MCrsCBbk1DZOMvgXNQOQFUg2RCJ3FAyROobXas9lZ34NKVpUOCcOjRnRk9Ogu6KfXB5Hek_PDObLyvGf7Jv7NqnyoO9BWZ_eaI3huksti-22T8PFy

I’m in. How do I test a chatbot?...

We feel really great about the future of chatbots and their ability to improve customer experience and to deliver higher quality leads. That being said, before jumping in with both feet we suggest running a chatbot experiment to validate or invalidate whether it works well for customers as well as your team.

Run this chatbot experiment.

We’ve organized a super simple experiment to help sales teams see if a chatbot could work for them:

  • Step 1: Develop a single hypothesis about what the chatbot will deliver
  • Step 2: Explicitly Identify the metric that will help you validate or invalidate your hypothesis
  • Step 3: Get benchmark data for that metric. You may be able to pull this from your current process or will need to build time into your experiment to capture it.
  • Step 4: Test chatbot
  • Step 5: Compare the two data sets to see if your hypothesis is valid or invalid.

Here is an example experiment:


Step 1 - Hypothesis: Implementing a chatbot will decrease the amount of time it takes to make first contact with a lead.

Step 2 - Metric: Hours from form filled to first contact.

Step 3 - Benchmark data: Depending on the size of your sales team, pull the data from at least 10% of your sales agents. If you are already measuring this metric, awesome! Just pull it and proceed to Step 4. If not, spend 10 days (without the agents’ knowledge) measuring this.

Step 4 - Test: There are a number of chatbot solutions out there. We recommend Intercom’s 14-day free trial because it’s a lean and easy way to get started and they have great analytics. Have the same agents you used for benchmark data spend 10 days using Intercom’s chatbot to qualify and make initial contact with leads.  

Step 5 - Compare: In a spreadsheet, take the data from your two sets of 10 days and compare your metric, ‘hours from form filled to first contact’. Which one has a lower average? Does one have a better conversion rate?


If you’re interested in testing out a chatbot for your sales, customer success, or customer support team, we hope this experiment helps. We’re also always available to talk through ideas and implement an experiment tailored to your team and goals. Just give us a shout!

Continued reading about Chatbots:

More Posts