[MUSIC] Our next step is to evaluate the sentiment model that we've built. Now, we just talked in this module
about classification error, precision, sorry,
false positives and false negatives. And we're gonna explore
that idea right here in this demonstration in this notebook. So the sentiment model has
a function called Evaluate. And that function Evaluate allows you to
evaluate its quality on some test data. And we're gonna provide
a particular metric and this metric is called the roc_curve. And we're gonna learn a little bit
more about the roc_curve next. But the roc_curve is a way to
explore the false positives and false negatives in that confusion
matrix that we discussed. Now, here is, it shows you the results of
evaluations that are hard to see in text. So let's do a little visualization of it,
again, using Canvas. So we can use sentiment_model.show, and we're gonna show the view
that we're gonna use, is going to be the Evaluation view. And here we go. Okay, this is really cool. We've built a few things. So this is a precision recall, I'm sorry,
this is what is called an roc_curve, it's a curve that trades off false
positives with true positives. Let me explain that a little bit. But first,
let's look at these numbers over here. So this is the confusion matrix. The number of true positives, the things that we got right
that were true, is 26,455. So right here. The number of true negatives
was only 4,000, so 3,965. So this is a highly unbalanced case. And then,
the number of false positives and the number of false
negatives is about the same. The overall accuracy was 91.1%, so 0.911. And also discuss a few metrics like
precision recall and false count which we're gonna learn more about later
in the course and later in specialization. Now, the thing about the false
positives and false negatives is that there's a very natural way to change
the threshold of what we believe the transition from negative class
to positive class should be. So this is what the threshold
shows on the right, but let me first show it on the curve here. So, you can see it is possible for
me to get, for example, a very high true positive rate. So .4, if I, sorry, a very low .4. If I dont' allow myself to
get any false positives. So, if I'm very worried about false
positives that can be very conservative, get no false positives. But again not capture many of the true
positives and make other mistakes. And you see. As you go through the curve
there's a kink here. This is the point where you get
not too many false positives but a lot of true positives. And then over on this end
you get every true positive. But you get a lot of false positives. How do you get every true positive but
get less false positives? You just say every data point is positive. Then you get all the true positives,
but you also make a lot of mistakes. And, we'll discuss more of this when
we talk about precision recall curve. But, nicely, we can change the threshold
over here, and walk that curve up there. As you can see, and so for example,
the far end, on the right, you see that the true
positive rate is really high. There's no false positives, but
that there's no true negatives. So there's no false negatives, no true
negatives, you just got everything right. And then you can slide
it just the other way. So you can play with this and really kind of get a good sense of what
this curve really means, the roc_curve. And that's the evaluation
that we're gonna be doing. [MUSIC]