Sign in or start a free trial to avail of this feature.
1. Introduction to AB Testing
In this lesson, you will be introduced to the concept of AB Testing and some common use cases for this technique.
What is AB testing?
- AB testing is an experiment where two versions of a variable, generally a control group and a treatment group, are compared against each other
Control and Treatment Groups
- A control group is a subset of a larger dataset that accurately reflects the qualities of the dataset as a whole
- A control group is often known as a baseline measure
- A treatment group is a subset of a larger dataset that manipulated in order to offer a comparison against the baseline
- It is good practice to ensure that the members of the treatment group also accurately reflect the qualities of the dataset as a whole before members of the group are manipulated
Some Uses of AB testing
- Determine ideal design for a website or retail space
- Test minor changes to a product
- Tailor products to specific customer segments
In this course, we'll look at AB testing and market basket analysis, a couple of more advanced modeling techniques that are widely used by businesses today.
We'll start with AB testing.
AB testing is a relatively simple concept where an experiment is performed on a subset of a larger population and then compared with an unchanged or control group.
AB testing is a key feature of website optimization as it's relatively easy to make small changes and then tabulate the results.
For example, an e-commerce business may present a subtly redesigned registration form to every 10th website visitor.
This allows them to test whether these users are more or less likely to complete the registration process.
If the answer is yes, then the redesigned form can be implemented more widely.
Due to the potentially large numbers involved, this small change can have a meaningful effect on the bottom line over time.
Alternatively, if the tests users react poorly to the redesign, the experiment can be halted without causing significant damage to the wider business.
Many businesses use AB testing to assist in evidence-based decision making.
It's particularly useful in cases where tastes and preferences are constantly evolving, and small changes are relatively cheap.
However, there is much more to AB testing than simply alternating the color of a landing page, and it should be thought of as a real-life hypothesis test.
In many industries, change requires a significant upfront investment with uncertain returns.
In these cases, a statistically rigorous sample group is crucial to deriving actionable conclusions.
Over the course of the following lessons, we'll use Alteryx's AB testing tools to inform a business investment decision.
Specifically, we'll dive into a case in which we're advising the managing director of Ben's Beverages, a chain of liquor stores operating in the UK.
Ben's Beverages specializes in selling wines, beers, and spirits.
They've become an established brand operating 123 stores. However, in recent years Ben's Beverages has come under increasing competition from supermarkets.
The managing director believes the business needs to do more to differentiate itself and revamp the store layout to improve the customer experience.
They've come to us for advice on how to approach this ambitious project.
They would like to implement changes across 20 stores and assess the results after a six-week trial.
The key metrics are total sales and footfall as measured by the total number of individual sales receipts.
If the revamped stores outperform the control group by at least 4%, the company will press ahead with a wider rollout.
We have historic data for all 123 stores over the past 12 months.
As we can see, this data set includes sales information and total receipt data for each day, along with fields identifying the day of the week, whether the store was open, and if the store is running a promotion.
We'll use this data to select an appropriate test group, also known as a treatment group.
These stores will be remodeled according to the new design.
We'll then compare the year-on-year performance of the test group against the control group, taking into account the underlying trends and seasonality in the business. Let's run through a quick summary of the high level steps for our AB analysis.
First, we'll use the historic data to identify a set of treatment stores for the new layout.
Next, we'll assign control stores to each treatment store to use as a comparison in our trial.
Finally, we'll analyze the results of the trial to see if the new layout led to increased sales and footfall.
We'll begin by preparing our historic data for analysis in the next lesson.