Sign in or start a free trial to avail of this feature.
1. Introduction to AB Testing
In this lesson, you will be introduced to the concept of AB Testing and some common use cases for this technique.
Overview of AB Testing (00:04)
AB Testing is a modeling technique where an experiment is performed on a subset of a larger population, and the results are compared with an unchanged control group.
AB testing is commonly used to optimize websites. In an AB test, some proportion of users will be shown a website element that is slightly different from normal, for example a redesigned registration form. The company can then analyze if people who see the new form are more or less likely to complete the registration process than people who see the normal form. If the new form is successful, it can be rolled out more widely, otherwise it can be withdrawn without significantly impacting the wider business.
AB testing can help assist in evidence-based decision making. It enables companies to make small and frequent changes, so it’s useful where preferences evolve quickly, and where small changes can be made quickly and cheaply.
Course Goal (01:29)
In many industries, change requires significant investment, and has uncertain returns. AB testing can be used to trial a business change on a small scale before rolling it out across the business. In our course, we use AB testing in Alteryx to inform a business investment decision.
Course Case Study (02:01)
Ben’s Beverages is a chain of liquor stores in the UK. The company has 123 stores, but is facing increased competition from supermarkets. They want to redesign their store layouts to improve the customer experience. The company intends to trial these changes in 20 stores for six weeks, then analyze the impact of the changes. The key metrics are total sales and footfall. If the revamped stores outperform the control group by 4%, the changes will be rolled out across all stores.
Our dataset contains daily sales data for all 123 stores over the last 12 months. We use this data to identify the treatment group, which is the group of stores that will be changed. The stores that will remain unchanged are called the control group.
High Level Steps for AB Analysis (03:41)
There are three high-level steps in an AB analysis:
Use historic data to identify treatment stores
Assign control stores to each treatment store to use as a comparison
Analyze the results of the trial and the impact on sales and footfall