Overview

In this lesson, you will be introduced to the concept of AB Testing and some common use cases for this technique.

To explore more Kubicle data literacy subjects, please refer to our full library.

Summary

  1. Overview of AB Testing (00:04)

    AB Testing is a modeling technique where an experiment is performed on a subset of a larger population, and the results are compared with an unchanged control group.

    AB testing is commonly used to optimize websites. In an AB test, some proportion of users will be shown a website element that is slightly different from normal, for example a redesigned registration form. The company can then analyze if people who see the new form are more or less likely to complete the registration process than people who see the normal form. If the new form is successful, it can be rolled out more widely, otherwise it can be withdrawn without significantly impacting the wider business.

    AB testing can help assist in evidence-based decision making. It enables companies to make small and frequent changes, so it’s useful where preferences evolve quickly, and where small changes can be made quickly and cheaply.

  2. Course Goal (01:29)

    In many industries, change requires significant investment, and has uncertain returns. AB testing can be used to trial a business change on a small scale before rolling it out across the business. In our course, we use AB testing in Alteryx to inform a business investment decision.

  3. Course Case Study (02:01)

    Ben’s Beverages is a chain of liquor stores in the UK. The company has 123 stores, but is facing increased competition from supermarkets. They want to redesign their store layouts to improve the customer experience. The company intends to trial these changes in 20 stores for six weeks, then analyze the impact of the changes. The key metrics are total sales and footfall. If the revamped stores outperform the control group by 4%, the changes will be rolled out across all stores.

    Our dataset contains daily sales data for all 123 stores over the last 12 months. We use this data to identify the treatment group, which is the group of stores that will be changed. The stores that will remain unchanged are called the control group.

  4. High Level Steps for AB Analysis (03:41)

    There are three high-level steps in an AB analysis:

    1. Use historic data to identify treatment stores

    2. Assign control stores to each treatment store to use as a comparison

    3. Analyze the results of the trial and the impact on sales and footfall 

Transcript

In this course, we'll look at AB Testing. This is an advanced modeling technique that's widely used by businesses today.

AB testing is a relatively simple concept where an experiment is performed on a subset of a larger population, and then compared with an unchanged or control group.

AB testing is a key feature of website optimization as it's relatively easy to make small changes and then tabulate the results.

For example, an e-commerce business may present a subtly redesigned registration form to every 10th website visitor.

This allows them to test whether these users are more or less likely to complete the registration process.

If the answer is yes, then a redesigned form can be implemented more widely.

Due to the potentially large numbers involved, this small change can have a meaningful effect on the bottom line over time.

Alternatively, if the test users react poorly to the redesign, the experiment can be halted without causing significant damage to the wider business.

Many businesses use AB testing to assist in evidence-based decision making.

It's particularly useful in cases where taste and preferences are constantly evolving, and small changes are relatively cheap.

However, there's much more to AB testing than simply alternating the color of a landing page and it should be thought of as a real life hypothesis test.

In many industries, change requires a significant upfront investment with uncertain returns.

In these cases, a statistically rigorous sample group is crucial to deriving actionable conclusions.

Over the course of the following lessons, we'll use Alteryx's AB Testing tools to inform a business investment decision.

Specifically, we'll dive into a case in which we're advising the managing director of Ben's beverages, a chain of liquor stores operating in the UK.

Ben's Beverages specializes in selling wines, beers, and spirits.

They've become an established brand operating 123 stores. However, in recent years, Ben's Beverages has come under increasing competition from supermarkets.

The managing director believes that business needs to do more to differentiate itself and revamp the store layout to improve the customer experience.

They've come to us for advice on how to approach this ambitious project.

They would like to implement changes across 20 stores and assess the results after a six-week trial.

The key metrics are total sales, and footfall as measured by the total number of individual sales receipts.

If the revamp stores outperformed the control group by at least 4%, the company will press ahead with the wider rollout.

We have historic data for all 120 stores over the past 12 months.

As we can see, this dataset includes sales information and total receipt data for each day.

Along with fields identifying the day of the week, whether the store was open, and if the store was running a promotion.

We'll use this data to select an appropriate test group also known as a treatment group.

These stores will be remodeled according to the new design.

We'll then compare the year on your performance of the treatment group against the control group, taking into account the underlying trends and seasonality in the business.

Let's run through a quick summary of the high level steps for our AB analysis.

First, we use the historic data to identify a set of treatment stores for the new layout.

Next, we'll assign control stores to each treatment store to use as a comparison in our trial.

Finally, we'll analyze the results of the trial to see if the new layout led to increase sales and footfall.

We'll begin by preparing our historic data for analysis in the next lesson.

Contents

My Notes

You can take notes as you view lessons.

Sign in or start a free trial to avail of this feature.

Free Trial

Download our training resources while you learn.

Sign in or start a free trial to avail of this feature.

Free Trial