In the previous video, we talked about evaluation metrics.
In this video, I'd like
to switch tracks a bit and
touch on another important aspect of
machine learning system design,
which will often come up, which
is the issue of how much
data to train on.
Now, in some earlier videos, I
had cautioned against blindly
going out and just spending
lots of time collecting lots of
data, because it's only
sometimes that that would actually help.
But it turns out that under
certain conditions, and I
will say in this video what those
conditions are, getting a
lot of data and training on
a certain type of learning
algorithm, can be a
very effective way to get
a learning algorithm to do very good performance.
And this arises often enough
that if those conditions hold true
for your problem and if
you're able to get a lot
of data, this could be
a very good way to get
a very high performance learning algorithm.
So in this video, let's
talk more about that.
Let me start with a story.
Many, many years ago, two researchers
that I know, Michelle Banko and
Eric Broule ran the following fascinating study.
They were interested in studying the
effect of using different learning
algorithms versus trying them
out on different training set sciences,
they were considering the problem
of classifying between confusable words,
so for example, in the sentence:
for breakfast I ate, should it be to, two or too?
Well, for this example,
for breakfast I ate two, 2 eggs.
So, this is one example
of a set of confusable words and that's a different set.
So they took machine learning
problems like these, sort of supervised learning
problems to try to categorize
what is the appropriate word to
go into a certain position
in an English sentence.
They took a few different learning
algorithms which were, you know,
sort of considered state of
the art back in the day,
when they ran the study in
2001, so they took a
variance, roughly a variance
on logistic regression called the Perceptron.
They also took some of
their algorithms that were fairly
out back then but somewhat less
used now so when the
algorithm also very similar
to which is a regression
but different in some ways, much
used somewhat less, used
not too much right now
took what's called a memory based
learning algorithm again used somewhat less now.
But I'll talk a little bit about that later.
And they used a naive based
algorithm, which is something they'll
actually talk about in this course.
The exact algorithms of these details aren't important.
Think of this as, you know, just picking
four different classification algorithms and really the exact algorithms aren't important.
But what they did was they
varied the training set size
and tried out these learning
algorithms on the range of
training set sizes and that's the result they got.
And the trends are very
clear right first most of
these outer rooms give remarkably similar performance.
And second, as the training
set size increases, on the
horizontal axis is the
training set size in millions
go from you know a
hundred thousand up to a
thousand million that is a
billion training examples. The
performance of the algorithms
all pretty much monotonically increase
and the fact that if
you pick any algorithm may be
pick a "inferior algorithm" but
if you give that "inferior
algorithm" more data, then from
these examples, it looks like
it will most likely beat even a "superior algorithm".
So since this original study
which is very influential, there's been
a range of many different
studies showing similar results
that show that many different learning
algorithms you know tend
to, can sometimes, depending on
details, can give pretty similar ranges
of performance, but what can
really drive performance is you can give the algorithm a ton of training data.
And this is, results like these
has led to a saying in
machine learning that often in
machine learning it's not
who has the best algorithm that
wins, it's who has the
most data So when is this
true and when is this not true?
Because we have a learning
algorithm for which this is
true then getting a
lot of data is often
maybe the best way to ensure
that we have an algorithm with
very high performance rather than
you know, debating worrying about exactly which of these items to use.
Let's try to lay out a
set of assumptions under which having
a massive training set we think will be able to help.
Let's assume that in our
machine learning problem, the features
x have sufficient information with which
we can use to predict y accurately.
For example, if we take
the confusable words all of them that we had on the previous slide.
Let's say that it features x
capture what are the surrounding
words around the blank that we're trying to fill in.
So the features capture then we
want to have, sometimes for breakfast I have black eggs.
Then yeah that is pretty
much information to tell me
that the word I want
in the middle is TWO and that
is not word TO and its not the word TOO. So
the features capture, you know, one
of these surrounding words then that
gives me enough information to pretty
unambiguously decide what is
the label y or in
other words what is the word
that I should be using to fill
in that blank out of
this set of three confusable words.
So that's an example what
the future ex has sufficient information
for specific y. For
a counter example.
Consider a problem of predicting
the price of a house from
only the size of the
house and from no other
features. So
if you imagine I tell you
that a house is, you
know, 500 square feet but I don't give you any other features.
I don't tell you that the
house is in an expensive part of the city.
Or if I don't tell you that
the house, the number of
rooms in the house, or how
nicely furnished the house
is, or whether the house is new or old.
If I don't tell you anything other
than that this is a
500 square foot house, well there's
so many other factors that would
affect the price of a
house other than just the size
of a house that if all
you know is the size, it's actually
very difficult to predict the price accurately.
So that would be a counter
example to this assumption
that the features have sufficient information
to predict the price to the desired level of accuracy.
The way I think about testing
this assumption, one way I
often think about it is, how often I ask myself.
Given the input features x,
given the features, given the
same information available as well as learning algorithm.
If we were to go to human expert in this domain.
Can a human experts actually or
can human expert confidently predict
the value of y. For this
first example if we go
to, you know an expert human English speaker.
You go to someone that
speaks English well, right, then
a human expert in English
just read most people like
you and me will probably we
would probably be able to
predict what word should go in
here, to a good English
speaker can predict this well,
and so this gives me confidence
that x allows us to predict
y accurately, but in contrast
if we go to an expert in human prices.
Like maybe an expert realtor, right, someone
who sells houses for a living.
If I just tell them the
size of a house and I
tell them what the price
is well even an expert
in pricing or selling
houses wouldn't be able
to tell me and so this is fine that
for the housing price example knowing
only the size doesn't give
me enough information to predict
the price of the house.
So, let's say, this assumption holds.
Let's see then, when having
a lot of data could help.
Suppose the features have enough
information to predict the
value of y.
And let's suppose we use a
learning algorithm with a
large number of parameters so
maybe logistic regression or linear
regression with a large number of features.
Or one thing that I sometimes
do, one thing that I often
do actually is using neural network with many hidden units.
That would be another learning
algorithm with a lot of parameters.
So these are all powerful learning
algorithms with a lot of parameters that
can fit very complex functions.
So, I'm going to call these, I'm
going to think of these as
low-bias algorithms because you
know we can fit very complex functions
and because we have
a very powerful learning algorithm,
they can fit very complex functions.
Chances are, if we
run these algorithms on
the data sets, it will
be able to fit the training
set well, and so
hopefully the training error will be slow.
Now let's say, we use
a massive, massive training set,
in that case, if we
have a huge training set, then
hopefully even though we have a lot of parameters
but if the training set is sort of even much
larger than the number of
parameters then hopefully these
albums will be unlikely to overfit.
Right because we have such a
massive training set and by
unlikely to overfit what that
means is that the training
error will hopefully be
close to the test error.
Finally putting these two
together that the train
set error is small and
the test set error is close
to the training error what
this two together imply is
that hopefully the test set error
will also be small.
Another way to
think about this is that
in order to have a high
performance learning algorithm we want
it not to have high bias and not to have high variance.
So the bias problem we're going
to address by making sure we
have a learning algorithm with many
parameters and so that
gives us a low bias alorithm
and by using a
very large training set, this ensures
that we don't have a variance problem here.
So hopefully our algorithm will
have no variance and so
is by pulling these two together,
that we end up with a low
bias and a low variance
learning algorithm and this
allows us to do well
on the test set.
And fundamentally it's a key ingredients
of assuming that the features
have enough information and we
have a rich class of functions
that's why it guarantees low bias,
and then it having a massive
training set that that's what guarantees more variance.
So this gives us a
set of conditions rather hopefully
some understanding of what's the
sort of problem where if
you have a lot of data
and you train a learning
algorithm with lot of parameters, that might
be a good way to give
a high performance learning algorithm
and really, I think the key test that
I often ask myself are
first, can a human experts
look at the features x and
confidently predict the value of
y. Because that's sort of
a certification that y
can be predicted accurately from
the features x and second,
can we actually get a large
training set, and train the
learning algorithm with a lot of
parameters in the training
set and if you can't do both
then that's more often give
you a very kind performance learning algorithm.