The term human-level
performance is sometimes used casually in research articles. But let me show you how we can
define it a bit more precisely. And in particular, use the definition
of the phrase, human-level performance, that is most useful for helping you drive
progress in your machine learning project. So remember from our last video that
one of the uses of this phrase, human-level error, is that it gives
us a way of estimating Bayes error. What is the best possible
error any function could, either now or in the future,
ever, ever achieve? So bearing that in mind, let's look at
a medical image classification example. Let's say that you want to look
at a radiology image like this, and make a diagnosis
classification decision. And suppose that a typical human,
untrained human, achieves 3% error on this task. A typical doctor, maybe a typical
radiologist doctor, achieves 1% error. An experienced doctor does even better,
0.7% error. And a team of experienced doctors, that is
if you get a team of experienced doctors and have them all look at the image and
discuss and debate the image, together their consensus
opinion achieves 0.5% error. So the question I want to pose to you is,
how should you define human-level error? Is human-level error 3%, 1%, 0.7% or 0.5%? Feel free to pause this video
to think about it if you wish. And to answer that question, I would urge
you to bear in mind that one of the most useful ways to think of human error is as
a proxy or an estimate for Bayes error. So please feel free to pause this video to
think about it for a while if you wish. But here's how I would
define human-level error. Which is if you want a proxy or
an estimate for Bayes error, then given that a team of
experienced doctors discussing and debating can achieve 0.5% error, we know that Bayes error
is less than equal to 0.5%. So because some system, team of these
doctors can achieve 0.5% error, so by definition, this directly,
optimal error has got to be 0.5% or lower. We don't know how much better it is,
maybe there's a even larger team of even more experienced doctors
who could do even better, so maybe it's even a little
bit better than 0.5%. But we know the optimal error
cannot be higher than 0.5%. So what I would do in this setting is use
0.5% as our estimate for Bayes error. So I would define human-level
performance as 0.5%. At least if you're hoping to use
human-level error in the analysis of bias and variance as we saw in the last video. Now, for the purpose of publishing
a research paper or for the purpose of deploying a system,
maybe there's a different definition of human-level error that
you can use which is so long as you surpass the performance
of a typical doctor. That seems like maybe a very
useful result if accomplished, and maybe surpassing a single radiologist,
a single doctor's performance might mean the system is good
enough to deploy in some context. So maybe the takeaway from this is to
be clear about what your purpose is in defining the term human-level error. And if it is to show that you can surpass
a single human and therefore argue for deploying your system in some context,
maybe this is the appropriate definition. But if your goal is the proxy for
Bayes error, then this is the appropriate definition. To see why this matters,
let's look at an error analysis example. Let's say, for
a medical imaging diagnosis example, that your training error is 5% and
your dev error is 6%. And in the example from the previous
slide, our human-level performance, and I'm going to think of this
as proxy for Bayes error. Depending on whether you defined it
as a typical doctor's performance or experienced doctor or team of doctors,
you would have either 1% or 0.7% or 0.5% for this. And remember also our definitions
from the previous video, that this gap between Bayes error or
estimate of Bayes error and training error is calling that
a measure of the avoidable bias. And this as a measure or an estimate of
how much of a variance problem you have in your learning algorithm. So in this first example,
whichever of these choices you make, the measure of avoidable bias
will be something like 4%. It will be somewhere between I guess, 4%, if you take that to 4.5%,
if you use 0.5%, whereas this is 1%. So in this example, I would say, it doesn't really matter which of the
definitions of human-level error you use, whether you use the typical
doctor's error or the single experienced doctor's error or
the team of experienced doctor's error. Whether this is 4% or 4.5%, this is
clearly bigger than the variance problem. And so in this case, you should focus on bias reduction
techniques such as train a bigger network. Now let's look at a second example. Let's see your training error is 1% and
your dev error is 5%. Then again it doesn't really matter,
seems but academic whether the human-level
performance is 1% or 0.7% or 0.5%. Because whichever of these definitions
you use, your measure of avoidable bias will be, I guess somewhere between
0% if you use that, to 0.5%, right? That's the gap between the human-level
performance and your training error, whereas this gap is 4%. So this 4% is going to be much bigger
than the avoidable bias either way. And so they'll just suggest you should
focus on variance reduction techniques such as regularization or
getting a bigger training set. But where it really matters will
be if your training error is 0.7%. So you're doing really well now,
and your dev error is 0.8%. In this case, it really matters that you
use your estimate for Bayes error as 0.5%. Because in this case, your measure of
how much avoidable bias you have is 0.2% which is twice as big as your measure for
your variance, which is just 0.1%. And so this suggests that maybe both the
bias and variance are both problems but maybe the avoidable bias is
a bit bigger of a problem. And in this example, 0.5% as we discussed
on the previous slide was the best measure of Bayes error, because a team of human
doctors could achieve that performance. If you use 0.7 as your proxy for
Bayes error, you would have estimated avoidable bias as pretty much 0%,
and you might have missed that. You actually should try to do
better on your training set. So I hope this gives a sense also of why
making progress in a machine learning problem gets harder as you achieve or
as you approach human-level performance. In this example,
once you've approached 0.7% error, unless you're very careful
about estimating Bayes error, you might not know how far
away you are from Bayes error. And therefore how much you should
be trying to reduce aviodable bias. In fact, if all you knew was that a single
typical doctor achieves 1% error, and it might be very difficult to know if you
should be trying to fit your training set even better. And this problem arose only when you're
doing very well on your problem already, only when you're doing 0.7%, 0.8%,
really close to human-level performance. Whereas in the two examples on the left,
when you are further away human-level performance, it was easier to target
your focus on bias or variance. So this is maybe an illustration of why
as your pro human-level performance is actually harder to tease out the bias and
variance effects. And therefore why progress on your machine
learning project just gets harder as you're doing really well. So just to summarize
what we've talked about. If you're trying to understand bias and
variance where you have an estimate of human-level error
for a task that humans can do quite well, you can use human-level error as a proxy
or as a approximation for Bayes error. And so the difference between your
estimate of Bayes error tells you how much avoidable bias is a problem,
how much avoidable bias there is. And the difference between
training error and dev error, that tells you how much variance is
a problem, whether your algorithm's able to generalize from the training
set to the dev set. And the big difference between
our discussion here and what we saw in an earlier course was that
instead of comparing training error to 0%, And just calling that
the estimate of the bias. In contrast, in this video we have a more
nuanced analysis in which there is no particular expectation that
you should get 0% error. Because sometimes Bayes error is non zero
and sometimes it's just not possible for anything to do better than
a certain threshold of error. And so in the earlier course,
we were measuring training error, and seeing how much bigger
training error was than zero. And just using that to try to
understand how big our bias is. And that turns out to work just fine for
problems where Bayes error is nearly 0%, such as recognizing cats. Humans are near perfect for that, so
Bayes error is also near perfect for that. So that actually works okay when
Bayes error is nearly zero. But for problems where the data is noisy,
like speech recognition on very noisy audio where it's just impossible
sometimes to hear what was said and to get the correct transcription. For problems like that,
having a better estimate for Bayes error can help you better
estimate avoidable bias and variance. And therefore make better decisions
on whether to focus on bias reduction tactics, or on variance reduction tactics. So to recap, having an estimate of
human-level performance gives you an estimate of Bayes error. And this allows you to more quickly make
decisions as to whether you should focus on trying to reduce a bias or trying to
reduce the variance of your algorithm. And these techniques will tend to work
well until you surpass human-level performance, whereupon you might no longer
have a good estimate of Bayes error that still helps you make this
decision really clearly. Now, one of the exciting developments
in deep learning has been that for more and more tasks we're actually able
to surpass human-level performance. In the next video, let's talk more about the process of
surpassing human-level performance.