[MUSIC]
We talked about accuracy and errors that a classifier might make.
But there are different kinds of errors.
So this kind of errors are called types of mistakes.
It's important to look at the types of mistakes a classifier might make.
And one way to do that is through what's called a confusion matrix.
So, let's take that in a little bit.
So we're talking about the relationship between the true label and
whatever classifier predicts, so the predicted label.
So let's say that if the true label is positive, and we predict a positive value
for that sentence, we call that a true positive because we got it right.
Similarly if the true label is negative and
we predict that negative we call that a true negative.
That's good cuz we got that right.
Now, there is two kinds of mistakes that I can make.
So, for example, if the true label's positive, but
we predicted as negative, we call that a false negative.
We said it was negative, but that was false cuz it's positive.
Similarly, if the true label is negative when we predicted as positive,
we call that a false positive.
It was negative, but we predicted it as positive.
And false positives and false negatives can have different impacts
on what can happen in practice with your classifier.
So let's look at a couple practical examples of that.
So let's look at two applications, and
what the cost has of false positives versus false negatives.
So, if you consider spam filtering, a false negative is
an email that was spam but went into my folder it thought it was not spam.
So that's just annoying I got another spam email in my inbox.
Maybe it's bad but not super bad.
However, if you look at a false positive
that's an email that was not spam that got labeled as spam, went to my spam filter.
I never saw it, I lost that email forever.
That has a higher cost.
Now we can also look at medical diagnosis or
other applications as a second application.
So what's a false negative in medical diagnosis?
False negative is, there's a disease that I have but
it didn't get detected, so the classifier said it was negative.
They don't have the disease.
So in this case, the disease goes untreated,
which can be a really bad thing.
But the false positives can also be a bad thing.
That is, I classify as having the disease when I never had the disease.
In this case I get treated potentially with a really bad drug or
false side effect for diseases that I never had.
So it's a little bit unclear what's worse, having a false positive or
a false negative.
In medialc complications it really depends on the cost of the treatment and
how many side effects it had versus how bad the disease can be.
Now this relationship between the true label and the predicted label,
false positive, false negatives, is called the Confusion Matrix.
This matrix we just do.
So for example, let's say that we have a setting with a 100 test examples.
And we have of those, 60 positive and 40 are negative.
So there's a little bit of class imbalance but not too much.
So of those 60 true positives, if I say I got 50 of them correct,
well of the 42 negatives
I got 35 of them correct.
Let's see what we've learned.
So out of the 100 examples I got 85 correct.
So we can talk about our accuracy.
Accuracy is 85 correct over 100, which is 0.85.
And we can also discuss the true positives and the true negatives.
Sorry, the false positives and the false negatives, so of the positives,
I got labeled as negative, that's a false negative.
And that was ten, I had ten false negatives and on the other hand,
of the true negatives we get five false positive.
So in this example, we got 85% accuracy.
We got a higher false negative rate, than we had a false positive rate.
Now those words, false positive,
false negative, apply only for minor classification for two classes.
But the ideal confusion matrix works well even when you have more classes.
So let's talk about a simple example of that.
So let's say that I have 100 test examples and this is for
medical diagnosis, so there's 3 classes, healthy, cold or flu.
And of the 100 test subjects we had 70 with that were healthy,
20 that had cold, and 10 that had the flu.
And let's suppose that we got 60 correct for
healthy, we got 12 correct for cold,
and we got 60, 12, 8 correct for flu.
So, the total, our accuracy, here,
was 80, which is 60 plus 12 plus 8 divided by 100.
So that 0.8, 80% accuracy.
But we can talk about the false predictions.
So from healthy there were ten mistakes.
And we can say it's more common to confuse healthy with having a cold
than it is with having the flu, because the flu is a more complex disease so
we might have those ten mistakes.
Eight were confused with code and two were confused with flu.
Cold can go both ways.
So we made eight mistakes.
Maybe you can say half of them got confused with healthy and
half of them got diagnosis something stronger the flu.
Well of the two mistakes for the flu, then maybe we say that we
made no mistakes often, nobody that came in for flu was thought oh you're healthy.
But two of those ten were thought to have just a cold and not the flu.
So this is an example of a confusion matrix, we can really
understand the types of mistakes we made and we can interpret those.
And this is a really important thing to do in classification