In this video, I'd like to tell you about learning curves.
Learning curves is often a very useful thing to plot.
If either you wanted to sanity check
that your algorithm is working correctly,
or if you want to improve the performance of the algorithm.
And learning curves is a
tool that I actually use
very often to try to
diagnose if a physical learning algorithm may be
suffering from bias, sort of variance problem or a bit of both.
Here's what a learning curve is.
To plot a learning curve, what
I usually do is plot
j train which is, say,
average squared error on my training
set or Jcv which is
the average squared error on my cross validation set.
And I'm going to plot
that as a function
of m, that is as a function
of the number of training examples I have.
And so m is usually a constant like maybe I just have, you know, a 100
training examples but what I'm
going to do is artificially with
use my training set exercise. So, I
deliberately limit myself to using only,
say, 10 or 20 or
30 or 40 training examples and
plot what the training error is and
what the cross validation is for this
smallest training set exercises.
So
let's see what these plots may look
like. Suppose I have only
one training example like that
shown in this this first example
here and let's say I'm fitting a quadratic function. Well, I
have only one training example. I'm
going to be able to fit it perfectly
right? You know, just fit the quadratic function. I'm
going to have 0
error on the one training example. If I
have two training examples. Well the quadratic function can also fit that very well. So,
even if I am using regularization,
I can probably fit this quite well.
And if I am using no neural regularization,
I'm going to fit this perfectly and
if I have three training examples
again. Yeah, I can fit a quadratic
function perfectly so if
m equals 1 or m equals 2 or m equals 3,
my training error
on my training set is
going to be 0 assuming I'm
not using regularization or it may
slightly large in 0 if
I'm using regularization and
by the way if I have
a large training set and I'm artificially
restricting the size of my
training set in order to J train.
Here if I set
M equals 3, say, and I
train on only three examples,
then, for this figure I
am going to measure my training error
only on the three examples that
actually fit my data too
and so even I have to say
a 100 training examples but if I want to plot what my
training error is the m equals 3. What I'm going to do
is to measure the
training error on the
three examples that I've actually fit to my hypothesis 2.
And not all the other examples that I have
deliberately omitted from the training
process. So just to summarize what we've
seen is that if the training set
size is small then the
training error is going to be small as well.
Because you know, we have a
small training set is
going to be very easy to
fit your training set
very well may be even
perfectly now say
we have m equals 4 for example. Well then
a quadratic function can be
a longer fit this data set
perfectly and if I
have m equals 5 then you
know, maybe quadratic function will fit to stay there so
so, then as my training set gets larger.
It becomes harder and harder to
ensure that I can
find the quadratic function that process through
all my examples perfectly. So
in fact as the training set size
grows what you find
is that my average training error
actually increases and so if you plot
this figure what you find
is that the training set
error that is the average
error on your hypothesis grows
as m grows and just to repeat when the intuition is that when
m is small when you have very
few training examples. It's pretty
easy to fit every single
one of your training examples perfectly and
so your error is going
to be small whereas
when m is larger then gets
harder all the training
examples perfectly and so
your training set error becomes
more larger now, how about the cross validation error.
Well, the cross validation is
my error on this cross
validation set that I haven't seen and
so, you know, when I have
a very small training set, I'm
not going to generalize well, just
not going to do well on that.
So, right, this hypothesis here doesn't
look like a good one, and
it's only when I get
a larger training set that,
you know, I'm starting to get
hypotheses that maybe fit
the data somewhat better.
So your cross validation error and
your test set error will tend
to decrease as your training
set size increases because the
more data you have, the better
you do at generalizing to new examples.
So, just the more data you have, the better the hypothesis you fit.
So if you plot j train,
and Jcv this is the sort of thing that you get.
Now let's look at what
the learning curves may look like
if we have either high
bias or high variance problems.
Suppose your hypothesis has high
bias and to explain this
I'm going to use a, set an
example, of fitting a straight
line to data that, you
know, can't really be fit well by a straight line.
So we end up with a hypotheses that maybe looks like that.
Now let's think what would
happen if we were to increase
the training set size. So if
instead of five examples like
what I've drawn there, imagine that
we have a lot more training examples.
Well what happens, if you fit a straight line to this.
What you find is that, you
end up with you know, pretty much the same straight line.
I mean a straight line that
just cannot fit this
data and getting a ton more data, well
the straight line isn't going to change that much.
This is the best possible straight-line
fit to this data, but the
straight line just can't fit this
data set that well. So,
if you plot across validation error,
this is what it will look like.
Option on the left, if you have already a miniscule training set size like you know,
maybe just one training example and is not going to do well.
But by the time you have
reached a certain number of training
examples, you have almost
fit the best possible straight
line, and even if
you end up with a much
larger training set size, a
much larger value of m,
you know, you're basically getting the same straight line,
and so, the cross-validation error
- let me label that -
or test set error or
plateau out, or flatten out
pretty soon, once you reached
beyond a certain the number
of training examples, unless you
pretty much fit the best possible straight line.
And how about training error?
Well, the training error will again be small.
And what you find
in the high bias case is
that the training error will end
up close to the cross
validation error, because you
have so few parameters and so
much data, at least when m is large.
The performance on the training
set and the cross validation set will be very similar.
And so, this is what your
learning curves will look like,
if you have an algorithm that has high bias.
And finally, the problem with
high bias is reflected in
the fact that both the
cross validation error and the
training error are high,
and so you end up with
a relatively high value of
both Jcv and the j train.
This also implies something very
interesting, which is that,
if a learning algorithm has high
bias, as we
get more and more training examples,
that is, as we move to
the right of this figure, we'll
notice that the cross
validation error isn't going
down much, it's basically fattened
up, and so if
learning algorithms are really suffering from high bias.
Getting more training data by
itself will actually not help
that much,and as our figure
example in the figure
on the right, here we had only five training.
examples, and we fill certain straight line.
And when we had a ton
more training data, we still
end up with roughly the same straight line.
And so if the learning algorithm
has high bias give me a lot more training data.
That doesn't actually help you
get a much lower cross validation
error or test set error.
So knowing if your learning
algorithm is suffering from high
bias seems like a useful
thing to know because this can
prevent you from wasting a
lot of time collecting more training
data where it might just not end up being helpful.
Next let us look at the
setting of a learning algorithm
that may have high variance.
Let us just look at the
training error in a around if
you have very smart training
set like five training examples shown on
the figure on the right and
if we're fitting say a
very high order polynomial,
and I've written a hundredth degree polynomial which
really no one uses, but just an illustration.
And if we're using a
fairly small value of lambda,
maybe not zero, but a fairly
small value of lambda, then
we'll end up, you know,
fitting this data very well that with
a function that overfits this.
So, if the training
set size is small, our training
error, that is, j train
of theta will be small.
And as this training set size increases
a bit, you know, we may
still be overfitting this
data a little bit but
it also becomes slightly harder to
fit this data set perfectly,
and so, as the training set size
increases, we'll find that
j train increases, because
it is just a little harder to fit
the training set perfectly when we have
more examples, but the training set error will still be pretty low.
Now, how about the cross validation error?
Well, in high variance
setting, a hypothesis is
overfitting and so the
cross validation error will remain
high, even as we
get you know, a moderate number
of training examples and, so
maybe, the cross validation
error may look like that.
And the indicative diagnostic that we
have a high variance problem,
is the fact that there's
this large gap between
the training error and the cross validation error.
And looking at this figure.
If we think about adding more
training data, that is, taking
this figure and extrapolating to
the right, we can kind
of tell that, you know the
two curves, the blue curve
and the magenta curve, are converging to each other.
And so, if we were to
extrapolate this figure to
the right, then it
seems it likely that the
training error will keep on
going up and the
cross-validation error would keep on going down.
And the thing we really care about is the cross-validation error
or the test set error, right?
So in this sort
of figure, we can tell that
if we keep on adding training
examples and extrapolate to the
right, well our cross validation
error will keep on coming down.
And, so, in the high
variance setting, getting more
training data is, indeed,
likely to help.
And so again, this seems like a
useful thing to know if your
learning algorithm is suffering
from a high variance problem, because
that tells you, for example that it
may be be worth your while
to see if you can go and get some more training data.
Now, on the previous slide
and this slide, I've drawn fairly
clean fairly idealized curves.
If you plot these curves for
an actual learning algorithm, sometimes
you will actually see, you know, pretty
much curves, like what I've drawn here.
Although, sometimes you see curves
that are a little bit noisier and
a little bit messier than this.
But plotting learning curves like
these can often tell
you, can often help you
figure out if your learning algorithm is
suffering from bias, or variance or even a little bit of both.
So when I'm
trying to improve the performance of
a learning algorithm, one thing
that I'll almost always do
is plot these learning
curves, and usually this will
give you a better sense of whether there is a bias or variance problem.
And in the next video
we'll see how this can
help suggest specific actions is
to take, or to not take,
in order to try to improve the performance of your learning algorithm.