7. Analyzing AB Trial Results

 
Subtitles Enabled

Sign up for a free trial to access more free content.

Free trial

Overview

In this final lesson on AB Testing, we will compare the performance of the treatment group with the control group over the trial period over various metrics.

Lesson Notes

Analyzing Trial Results

  • As per the Managing Director’s instruction, we will analyze the trial results to see if both sales and till receipts have risen by at least 4.5%

AB Analysis Tool

  • The AB Analysis tool compares the change in a user-defined value between two groups over a specified time period
  • The C input node should link to the datastream containing the list of control units
  • The T input node should link to the datastream containing the list of treatment units
  • The P input node should link to the datastream containing the performance data for all units, that is the data we want to measure
  • Each AB Analysis tool can only compare one performance measure, so we’ll need to use one tool to measure receipts and another to measure sales

Transcript

In the previous lesson, we consolidated the trial and historical sales data to prepare for our analysis of the Ben's Beverages store revamp. The managing director has specified that two important KPIs be considered. Sales, and footfall. Remember, Ben's Beverages will expand the new store format across its properties if the treatment group outperforms the control group by at least 4% on both of these measures.

Our goal in this lesson is to compare the performance of the treatment group versus the control group over the trial period. We'll achieve this goal through two key steps. First, we'll analyze the receipts to determine if the revamp resulted in any change to footfall.

Next, we'll analyze sales and deliver our final results.

We'll begin by considering change in footfall, using receipts as a proxy. We'll use an AB analysis tool to determine the change.

We'll navigate to the AB testing tab on the tools pallet and bring an AB analysis tool onto the canvas.

This tool has three inputs and must be handled carefully. The C input takes a work stream from the C output node of the AB controls tool and the AB prep container.

Remember, the AB controls tool has our control and treatment pairs.

The T node takes the work stream from the treatment group, which we separated using the summarize tool in the treatment group container.

Finally, the P node takes the results of the performance data, specifically the results from our trial. This is the work stream coming from the union tool in the trial sample groups container.

Let's now configure the AB analysis tool.

We'll navigate to the configuration window, specify the control unit identifier as controls, and the control unit to treatment unit mapping field as treatments.

The treatment unit identifier for the treatments will be store.

Finally, the performance unit identifier is again store, with the data at the weekly level focusing on the measure of sum receipts.

We'll now move on to the dates tab and ensure that our start and end dates are correct.

We'll set the start date as February 16th 2015 and the end date as March 29th 2015.

We'll then right click on the tool, choose to add all browses, and run the work flow.

Let's click on the first browse and open it in a new window.

The first two items in this window are the start and end date of the trial as configured and the test summary data. The summary tells us that the number of receipts for our treatment group increased by 6.3% versus the same period last year. However the control group only managed an increase of 1.7%.

Therefore the treatment group outperformed the control group by 4.5% in aggregate.

The average outperformance, or lift, was 4.8%.

The third section in this window, for lift analysis, gives us a little more detail on the average outperformance.

Scrolling down, section four breaks down the average, max, min and standard deviation of the respective groups.

Next we come to a chart that displays the treatment group stores and their respective control stores. We can see that in general, the blue dots, or treatment stores, performed significantly better than the control stores.

Below this chart, we have a line chart, a standard box and whisker plot, and a violin plot. We won't go through these charts, but feel free to review them in your own time. At this point, we're ready to move on to the second step and analyze the sales data. We'll bring a second AB analysis tool onto the canvas and again connect the C input to the AB controls work stream, the T input to the treatment group work stream, and the P input to the trial sample groups work stream.

We'll configure this tool in the same manner as the previous one. We'll again specify the control unit identifier as controls, the control unit as treatments, both the treatment unit identifier and performance unit identifier as store, and view performance data on the weekly level. This time, we'll specify the performance measure as sum sales. We'll then go to the dates tab, set our start date as February 16th 2015 and our end date as March 29th 2015.

Again, we'll add all browses and run the work flow.

Let's open the first browse tool and look at the results.

The summary tells us that year over year sales for the treatment group grew by 8.9% versus 4.3% for the controls.

Again, this is an outperformance of 4.5%. As you might expect, lift was again 4.8%. Based on the results from these two analyses, it's clear that the store revamp has resulted in a marked improvement well above the 4% threshold required by the managing director. We'll recommend that the new layout is rolled out to all Ben's Beverages stores. This marks an end to the AB testing section of this course. In the next lesson, we'll introduce the concept of market basket analysis.