[MUSIC]
Our next step is to evaluate
the sentiment model that we've built.
Now, we just talked in this module about classification error,
precision, sorry, false positives and false negatives.
And we're gonna explore that idea right here
in this demonstration in this notebook.
So the sentiment model has a function called Evaluate.
And that function Evaluate allows you to evaluate its quality on some test data.
And we're gonna provide a particular metric and
this metric is called the roc_curve.
And we're gonna learn a little bit more about the roc_curve next.
But the roc_curve is a way to explore the false positives and
false negatives in that confusion matrix that we discussed.
Now, here is, it shows you the results of evaluations that are hard to see in text.
So let's do a little visualization of it, again, using Canvas.
So we can use sentiment_model.show, and
we're gonna show the view that we're gonna use,
is going to be the Evaluation view.
And here we go.
Okay, this is really cool.
We've built a few things.
So this is a precision recall, I'm sorry, this is what is called an roc_curve,
it's a curve that trades off false positives with true positives.
Let me explain that a little bit.
But first, let's look at these numbers over here.
So this is the confusion matrix.
The number of true positives,
the things that we got right that were true, is 26,455.
So right here.
The number of true negatives was only 4,000, so 3,965.
So this is a highly unbalanced case.
And then, the number of false positives and
the number of false negatives is about the same.
The overall accuracy was 91.1%, so 0.911.
And also discuss a few metrics like precision recall and false count
which we're gonna learn more about later in the course and later in specialization.
Now, the thing about the false positives and false negatives is that
there's a very natural way to change the threshold of what we believe
the transition from negative class to positive class should be.
So this is what the threshold shows on the right, but
let me first show it on the curve here.
So, you can see it is possible for me to get, for
example, a very high true positive rate.
So .4, if I, sorry, a very low .4.
If I dont' allow myself to get any false positives.
So, if I'm very worried about false positives that can be very conservative,
get no false positives.
But again not capture many of the true positives and make other mistakes.
And you see.
As you go through the curve there's a kink here.
This is the point where you get not too many false positives but
a lot of true positives.
And then over on this end you get every true positive.
But you get a lot of false positives.
How do you get every true positive but get less false positives?
You just say every data point is positive.
Then you get all the true positives, but you also make a lot of mistakes.
And, we'll discuss more of this when we talk about precision recall curve.
But, nicely, we can change the threshold over here, and walk that curve up there.
As you can see, and so for example, the far end, on the right,
you see that the true positive rate is really high.
There's no false positives, but that there's no true negatives.
So there's no false negatives, no true negatives, you just got everything right.
And then you can slide it just the other way.
So you can play with this and
really kind of get a good sense of what this curve really means, the roc_curve.
And that's the evaluation that we're gonna be doing.
[MUSIC]