In this video, I'm going to talk about the history of backpropagation.
I'll start with where it came from in the 70s' and 80s,' and then I'll talk a bit
about why it failed in the 90s.' That is, why serious machine learning research has
abandoned it. There was a popular view of why this
happened, and we can now see that, that popular view was largely wrong.
The real reasons it was abandoned were because computers were too slow and data
sets were too small. I'll conclude by showing you a historical
document. There was a bet made between two machine
learning researchers in 1995. It's interesting to see what people back
then believed and how wrong they were. Backpropagation was invented independently
several times in the 70s' and 80s'. It started in the late 60s' with control
theories called Bryson and Ho who invented a linear version of backpropagation.
Paul Werbos went to their lectures and realized it could be made non-linear.
And in his thesis in 1974, he published what's probably the first proper version
of backpropogation. Rumelhart and Williams and I invented it
in 1981 without knowing about Paul Werbos's work.
But we tried it out, and it didn't work very well for the first thing we tried it
for, And so we abandoned it.
David Parker invented it in 1985, and so did Yann LeCun.
Also in 1985, I went back and tried again the thing that Rumelhart, Williams and I
had abandoned and discovered it worked pretty well.
In 1986, we produced a paper with a really convincing example of what it could do.
It was clear the backpropagation had a lot of promise for learning multiple layers of
non-linear future detectors. But it didn't really live up to its
promise. And by the late 1990s, most of the serious
researchers in machine learning had given up on backpropagation.
For example, in David Mukai's textbook, there's very little mention of it.
It was still widely used by psychologists for making psychological models.
And it was also quite widely used in practical applications, such as credit
card fraud detection. But in machine learning, people thought it
had been supplanted by support vector machines.
The popular explanation of what happened to backpropagation in the late 90s' was
that it couldn't make use of multiple layers and non-linear features.
This wasn't true in convolutional nets, which were the exception.
But in general, people couldn't get feed forward neural networks trained with
backpropagation to do impressive things if they had multiple hidden layers,
Except for some toy examples. It also did not work well in recurrent
networks or in deep auto-encoders, Which we'll cover in our later lecture.
And recurrent networks were perhaps the place where its most exciting, and so it
was there that it was most disappointing that people couldn't make it work well.
Support vector machines by contrast, worked well. They didn't require as much
expertise to make them work, they produced repeatable results, and they had a much
better theory. And they had much fancy of theory.
So, that was the popular explanation of what went wrong with backpropagation.
With a more historical prospective, we can see why really failed.
The computers were thousands of times too slow,
And the label data's sets hundreds of times too small for the regime in which
backpropagation would really shine. Also, the deep networks, as well as being
too small, were not sensibly initialized. And so, backpropagating through deep
networks didn't work well because the gradients tended to die, because the
initial weights were typically too small. These issues prevented backpropagation
from being successful. For tasks like vision and speech,
They would eventually be a big win. So, we need to distinguish between
different kinds of machine learning task. There's ones that are more typical of the
kinds of things people study in statistics,
And ones that are more typical of the kinds of things people study in artificial
intelligence. So, at the statistics end of the spectrum,
you typically have low dimensional data. A statistician thinks of a 100 dimensions
as high dimensional data. At the artificial intelligence end of the
spectrum, things like images or coefficients representing speech typically
have many more than a 100 dimensions. At the statistics end of the spectrum,
There's usually a lot of noise in the data.
Whereas, in the AI end of the spectrum, noise isn't the real problem.
For statistics, there's often not that much structure in the data, and what
structure there is can be captured by a fairly simple model.
At the AI end of the spectrum, there's typically a huge amount of structure in
the data. So if you take a set of images, its highly
structured data. But the structure is too complicated to be
captured by a simple model. So in statistics, the main problem is
separating true structure from noise, not thinking that noise is really structure.
This can be done by a Bayesian neural net pretty well.
But for typical non-Bayesian neural nets, it's not the kind of problem they're good
at. And so, for problems like that, it makes
sense to try a support vector machine or a method called a Gaussian process if you're
doing regression, which I'll talk about briefly later.
At the artificial intelligence end of the spectrum, the main problem is to find a
way of representing all this complicated structure so that it can be learned.
The obvious thing to do is to try to hand design appropriate representations.
But actually, it's easier to let back propagation figure out what
representations to use by giving it multiple layers and using a lot of
computation power to let it decide what the representation should be.
I now want to talk very briefly about support vector machines.
I'm not going to explain how they work, But I am going to say what I think their
limitations are. So, there's several ways in which you can
view a support vector machine, and I'm going to give you two different views of
them. According to the first view, support
vector machines are just the reincarnation of perceptions with a clever trick called
the kernel trick. So, the idea is that you take the inputs,
You expand the raw input into a very large lair of non-linear, but also non adaptive
features. So, that's just like perceptrons where you
have this big layer of features it doesn't learn.
And then, you only have to learn one layer of adaptive weights, the weights from the
features to the decision unit. And support vector machines have a very
clever way of avoiding overfitting when they learn those weights.
They look for what's called a maximum margin hyperplane in a high dimensional
space, And they can do that much more efficiently
than you might have thought possible. And, that's why they work well.
The second view also views support vector machines as a clever reincarnation of
perceptrons. But, it has a completely different notion
of what kinds of features they're using. So, according to the second view,
Each input vector in the training set is used to define one feature.
I'll spell it differently to indicate it's a completely different kind of feature
from the first kind. Each of these features gives a scale of
value which involves doing a global match between a test input and that particular
training input. So, roughly speaking, it's how similar the
test input is to a particular training case.
Then, there's a clever way of simultaneously finding how to weight those
features, so as to make the right decision,
And also during feature selection. That is, deciding which of those features
not to use. So, although these views sound extremely
different from one another, they're just two alternatives ways of looking at the
same thing, A support vector machine.
And, in both cases, It's using non-adaptive features and then
one layer of adaptive weights. And that limits to what you can do that
way. You can't learn multiple layers of
representation with a support vector machine.
This is a historical document from 1995. It was given to me by [unknown] and it's a
bet between Larry Jackel, who headed the adaptive systems research group at Bell
Labs, And Vladamir Vapnik, who is the leading
proponent of support vector machines. Larry Jackel bet that by 2000, people
would understand why big neural nets trained with backpropagation worked well
on large data sets. That is, they would understand it
theoretically in terms of conditions and bands.
Vapnik bet that they wouldn't, But he made a side bet.
That if he was the one to figure it out, he would win anyway.
Vapnik in turn, Bet that by 2005, nobody will be using big
neural nets like that train backpropagation.
It turns out that they were both wrong. The limitation to using big neural nets
with backpropagation was not that we didn't have a good theory and not that
they were essentially helpless, but that we didn't have big enough computers or big
enough data sets. It was a practical limitation not a
theoretical one.