6. Validating our Models

 
Subtitles Enabled

Sign up for a free trial to access more free content.

Free trial

Overview

In this lesson, you will learn how to validate various trained models and contrast their performance using a lift chart.

Lesson Notes

Lift Chart Tool

- The Lift Chart Tool compares predictions from different binary models, splitting data in each model into 10 groups

-  The tool offers cumulative response rate and incremental response rate charts

- The tool provides both visual and numeric representations of the model results

Transcript

In the previous lesson, we compared three different decision tree models to see if various of combinations of predictor variables would lead to greater accuracy. We found that our second and third models both offered improved accuracy over the initial model. However, these comparisons were against the dataset we used to create each of the models. In this lesson, we're going to compare each of the models' performance against the validation dataset, using the Lift Chart tool. We'll accomplish this goal in three key steps. First, we'll go back to the original dataset, and calculate the grant award percentage rate. Next, we'll configure a Lift Chart tool using that information.

Finally, we'll run the workflow and analyze the results.

In order to compare each of the models, we'll use a Lift Chart tool. However, to configure this Lift Chart tool correctly, we must first calculate the average rate of grants awarded. That is to say, the total number of grants, divided by the total number of applications. To calculate this metric, we'll start by bringing a Formula tool onto the canvas, and connecting to the Grant Application dataset.

We'll create a new field, called Grant Status Number.

Simply enter a tonumber formula for the Grant Status field, and change the datatype to Int16.

Next, we'll connect the Summarize tool to the Formula tool.

We'll set the tool to Sum the Grant Status Number field, and rename it to Number of Grants.

We'll also count the total number of applications, by referencing the same field, and renaming this field Applications.

We'll now connect the Formula tool to calculate the average grant rate. We'll create a new column, called Grant Rate, and simply divide the number of grants by applications.

We'll change the data type to FixedDecimal, size 12.3, as we would like three decimal places in our result. We'll know run the workflow. As with before, this could take some time, so I'll cut the wait time out of the video.

We'll look at the results of the Formula tool, and see that the grant rate of the entire dataset is 0.458, or 45.8 percent.

We'll use this figure in the Lift Chart tool. At this point, we're ready to move onto step two, and use a Lift Chart tool to compare the models against the validation set. Before we bring a Lift Chart tool onto the canvas, we'll need to brings medals together with a Union tool. To that end, we'll bring a Union tool onto the canvas, and connect the output node from each of the three Decision Tree tools. We're now ready to connect our Lift Chart. We'll bring a Lift Chart onto the canvas, and connect one input node to the Union tool.

The second input node will reference the validation dataset coming from the Create Samples tool. In the configuration window of the Lift Chart tool, we have the option to create a total cumulative response chart, or an incremental response chart. We'll choose total cumulative response for now.

In the true response rate field, we'll enter the grant rate of 0.458 we calculated previously.

We'll also set the target level to one. We'll now add a browse tool, and run the workflow.

Once the workflow finishes running, we'll be ready to move onto the final step, and analyze the results of the Lift Chart. We'll click on the Browse tool, and expand the window. We're presented with a chart contrasting the sample population with the total response captured. The sample population on the x-axis is the total number of grant applications in the validation dataset. The total response captured on the y-axis represents the number of grants awarded. The dark, black line dividing the chart is our grant rate of 0.458.

We can use that a hundred percent of our applications equates to a hundred percent of the grants awarded. In this case, it would mean that for every thousand applications, there were 458 grants awarded. The three colored lines above the grant rate line are the improvements over random chance achieved by our three models respectively. What we're looking for is a model which captures a high proportion of the grant status on a smaller population of sample. If we look at the fourth decile, or 0.4 on the x-axis, both Decision Trees Two and Three capture 70 percent of the responses, compared to Decision Tree One, which fares less well. If we scroll down, we can see a table that lays out this data in numerical format. Again, we can see at the fourth decile, both Decision Tree Two and Three actually capture 71 percent of predicted grant applications.

Notice at the second decile, Decision Tree Two has the upper-hand.

Faced with a decision between these three models, which one do we prefer? Well, Decision Tree Two requires 11 predictor variables, versus 22 for Decision Tree Three.

As a general rule, simpler models are better. Further, for smaller sample sizes, Decision Tree Two does at least as well as Decision Tree Three, if not better. As such, in this instance, we would prefer Decision Tree Two.

To recap, we compared the performance of our three models against a validation set using a Lift Chart tool. We achieved this in three key steps. First, we calculated the grant award rate, so that we could apply it to the Lift Chart tool. We then configured the Lift Chart tool to compare our models against the validation set. Finally, we ran the workflow, and analyzed the results, determining a preference for Decision Tree Two. We spent the last several lessons looking at decision tree tools. However, other predictive models may provide even better results. Before looking into other models, we'll take a deep dive into the confusion matrix in the next lesson.