We've talked about how to evaluate
learning algorithms, talked about model selection,
talked a lot about bias and variance.
So how does this help us figure
out what are potentially fruitful,
potentially not fruitful things to
try to do to improve the performance of a learning algorithm.
Let's go back to our original
motivating example and go for the result.
So here is our earlier example
of maybe having fit regularized
linear regression and finding that it doesn't work as well as we're hoping.
We said that we had this menu of options.
So is there some way to
figure out which of these might be fruitful options?
The first thing all of this
was getting more training examples.
What this is good for,
is this helps to fix high variance.
And concretely, if you instead
have a high bias problem and
don't have any variance problem, then
we saw in the previous video
that getting more training examples,
while maybe just isn't going to help much at all.
So the first option is useful
only if you, say, plot
the learning curves and figure
out that you have at least
a bit of a variance, meaning that
the cross-validation error is, you know,
quite a bit bigger than your training set error.
How about trying a smaller set of features?
Well, trying a smaller
set of features, that's again
something that fixes high variance.
And in other words, if you figure
out, by looking at learning curves
or something else that you used,
that have a high bias
problem; then for goodness
sakes, don't waste your time
trying to carefully select out
a smaller set of features to use.
Because if you have a high bias problem, using
fewer features is not going to help.
Whereas in contrast, if you
look at the learning curves or something
else you figure out that you
have a high variance problem, then,
indeed trying to select
out a smaller set of features,
that might indeed be a very good use of your time.
How about trying to get additional
features, adding features, usually,
not always, but usually we
think of this as a solution
for fixing high bias problems.
So if you are adding extra
features it's usually because
your current hypothesis is too
simple, and so we want
to try to get additional features to
make our hypothesis better able
to fit the training set. And
similarly, adding polynomial features;
this is another way of adding
features and so there
is another way to try
to fix the high bias problem.
And, if concretely if
your learning curves show you
that you still have a high
variance problem, then, you know, again this
is maybe a less good use of your time.
And finally, decreasing and increasing lambda.
This are quick and easy to try,
I guess these are less likely to
be a waste of, you know, many months of your life.
But decreasing lambda, you
already know fixes high bias.
In case this isn't clear to
you, you know, I do encourage
you to pause the video and think through this that
convince yourself that decreasing lambda
helps fix high bias, whereas increasing
lambda fixes high variance.
And if you aren't sure why
this is the case, do
pause the video and make
sure you can convince yourself that this is the case.
Or take a look at the curves
that we were plotting at the
end of the previous video and
try to make sure you understand
why these are the case.
Finally, let us take everything
we have learned and relate it back
to neural networks and so,
here is some practical
advice for how I usually
choose the architecture or the
connectivity pattern of the neural networks I use.
So, if you are fitting
a neural network, one option would
be to fit, say, a pretty
small neural network with you know, relatively
few hidden units, maybe just
one hidden unit. If you're fitting
a neural network, one option would
be to fit a relatively small
neural network with, say,
relatively few, maybe only one
hidden layer and maybe
only a relatively few number
of hidden units.
So, a network like this might have relatively
few parameters and be more prone to underfitting.
The main advantage of these small
neural networks is that the computation will be cheaper.
An alternative would be to
fit a, maybe relatively large
neural network with either more
hidden units--there's a lot
of hidden in one there--or with more hidden layers.
And so these neural networks tend
to have more parameters and therefore be more prone to overfitting.
One disadvantage, often not a
major one but something to
think about, is that if you have
a large number of neurons
in your network, then it can
be more computationally expensive.
Although within reason, this is often hopefully not a huge problem.
The main potential problem of
these much larger neural networks is that it could be more prone to overfitting
and it turns out if you're applying neural
network very often using
a large neural network often it's actually the larger, the better
but if it's overfitting, you can
then use regularization to address
overfitting, usually using
a larger neural network by using
regularization to address is
overfitting that's often more
effective than using a smaller neural network.
And the main possible disadvantage is
that it can be more computationally expensive.
And finally, one of the other decisions is, say,
the number of hidden layers you want to have, right?
So, do you want
one hidden layer or do
you want three hidden layers, as
we've shown here, or do you want two hidden layers?
And usually, as I
think I said in the previous
video, using a single
hidden layer is a reasonable default, but
if you want to choose the
number of hidden layers, one
other thing you can try is
find yourself a training cross-validation,
and test set split and try
training neural networks with one
hidden layer or two hidden
layers or three hidden layers and
see which of those neural
networks performs best on the cross-validation sets.
You take your three neural networks
with one, two and three hidden
layers, and compute the
cross validation error at Jcv
and all of
them and use that to
select which of these
is you think the best neural network.
So, that's it for
bias and variance and ways
like learning curves, who tried to diagnose these problems.
As far as what
you think is implied, for one
might be truthful or not
truthful things to try
to improve the performance of a learning algorithm.
If you understood the contents
of the last few videos and if
you apply them you actually
be much more effective already and
getting learning algorithms to work on problems
and even a large fraction,
maybe the majority of practitioners
of machine learning here in
Silicon Valley today doing these things as their full-time jobs.
So I hope that these
pieces of advice
on by experience in diagnostics
will help you to much effectively
and powerfully apply learning and
get them to work very well.