[MUSIC]
We've discussed error and accuracy as ways to evaluate a classifier.
Now, it's very important to understand the accuracies or
errors that you're actually getting from your classifier.
I really think deeply about whether those are good errors or
good levels of accuracy in your situation.
So for example, one of the common mistakes you might make
is to say how good is my classification at all?
When you build a classifier,
the first baseline comparison it should do is against random guessing.
So for example, if you have a binary classification problem like is this
sentence of positive or negative sentiment, then just random
guessing is gonna give you 50% accuracy on average, so you better beat 50%.
If you have k classes, so for example, if you have 3 classes.
You're gonna have a random guessing accuracy of 33%.
For 4 classes it would be 25%, for k classes it would be 1 over k.
So at the very least it should beat random guessing really well.
Because if you don't then your approach is basically pointless.
Now, even beyond beating random guessing, truly think deeply about whether you
classify, even it if looks really good, is it really meaningfully good?
So for example, suppose you have a span predictor that gets 90% accuracy.
Should you go brag about it?
Is that awesome?
Well, it really depends.
So the case of spam, not so good, because in 2010 data shows that 90%
of the emails ever sent were spam, 90% of the emails.
So if I just guess that every email is spam, what accuracy do I get?
90%.
This is a problem where this is what's called majority class prediction so
its just predicted classes most common.
And it can have amazing performance in cases where there's what's called class
imbalance.
One class has much more representation than the others.
Spam is much more representative than regular good emails.
And so, you have to be very cautious and really look at whether you have class
imbalance when you try to figure out whether your accuracy is good.
And of course, this also beats,
this approach also beats random guessing, if you know what majority class is.
So you should always be digging into your problem, and
understanding really thinking about the predictions you're getting and
whether that accuracy is really meaningfully good for your problem.
So ask yourself questions like, is there class imbalance?
How they compare against baseline approaches like random guessing,
majority class and really fancier things than that.
And most importantly, think about your application and
ask yourself, what is a good enough accuracy to make my users really happy?
So, in spam filtering, if your accuracy is not that good, then there'll be important
messages going to the spam folder, and that could be a really bad thing.
[MUSIC]