In this lecture, I'll introduce belief nets.
One of the reason I abandoned back propagation in the 1990's is because it
required too many labels. Back then, we just didn't have data sets
with sufficient numbers of labels. I was also influenced by the fact that
people managed to learn with very few explicit labels.
However, I didn't want to abandon the advantages of doing gradient descend
learning to learn a whole bunch of weights.
So the issue was, was there another objective function that we could do
grading decentive? The obvious place to look was generative
models where the objective function is to model the input data rather than
predicting a label. This meshed nicely with a major movement
in statistics and artificial intelligence called the graphical models.
The idea of graphical models was to combine discrete graph structures for
representing how variables depended on one another.
With real valued computations that inferred the probability of one variable,
given the observed values of other variables.
Boltzmann Machines were actually a very early example of a graphical model,
But they were undirected graphical models. In 1992, Radford Neal pointed out that
using the same kinds of units as we used in Boltzmann machines, we could make
directed graphical models which he called Sigmoid Belief Nets.
And the issue then became, how can we learn Sigmoid belief nets?
The second problem is that for deep networks, the learning time does not scale
well. When there's multiple hidden layers, the
learning was very slow. You might ask why this was,
And we now know that one of the reasons was we did not initialize the weights in a
sensible way. Yet, another problem is the back
propagation can get stuck in poor local optima.
These are often quite good, so back propagation is useful.
But we can now show that for deep nets, the local optima you get stuck in, if you
start with small random weights are typically far from optimal.
There is the possibility of retreating to simpler models that allow convex
optimization. But, I don't think this is a good idea.
Mathematicians like to do that because they can prove things.
But in practice, you're just running away from the complexity of real data.
So, one way to overcome the limits of back propagation is by using unsupervised
learning. The idea is that we want to keep the
efficiency and simplicity of using a gradient method and stochastic mini batch
descent for adjusting weights. But, we're going to use that method for
modeling the structure of the sensory input, not for modeling the relation
between input and output. So the idea is, the weights are going to
be adjusted to maximize the probability that a generative model would have
generated the sensory input. We already saw that in learning Boltzmann
machines. And one way to think about it is, if you
want to do computer vision, you should first learn to do computer graphics.
To first order, computer graphics works and computer vision doesn't.
The learning objective for a generative model, as we saw with Boltzmann machines,
is to maximize the probability of the observed data not to maximize the
probability of labels given inputs. Then the question arises, what kind of
generative model should we learn? We might learn an energy based model like
the Boltzmann machine, Or we might learn a causal model made of
idealized neurons, and that's what we'll look at first.
Well finally, we might learn some kind of hybrid of the two, and that's where we'll
end up. So, before I go into causal belief nets
made of neurons, I want to give you a little bit of background about artificial
intelligence and probability. In the 1970's and early 1980's, people in
artificial intelligence were unbelievably anti-probability.
When I was a graduate student, if you mentioned probability, it was assigned
that you were stupid and that you just hadn't got it.
Computers were all about discrete single processing, and if you'd introduce any
probabilities they would just infect everything.
It's hard to conceive of how much people are against probability, so here's a quote
to help you. I'll read it out.
Many ancient Greeks supported Socrates opinion that deep, inexplicable thoughts
came from the gods. Today's equivelant to those gods is the
erratic, even probabilistic neuron. It is more likely that increased
randomness of neural behavior is the problem of the epileptic and the drunk,
not the advantage of the brilliant. That was in Patrick Henry Winston's first
AI textbook, in the first edition. And it was the general opinion at the
time. Winston was to become the leader of the
MIT AI Lab. Here's an alternative view.
All of this will lead to theories of computation which are much less rigidly of
an all-or-none nature than past and present formal logic.
There are numerous indications to make us believe that this new system of formal
logic will move closer to another discipline which has been little linked in
the past with logic. This is thermodynamics, primarily in the
form it was received from Boltzmann. That was written by John von Neumann in
1957, and was part of the unfinished manuscript he left behind for what was to
be his crowning achievement, His book on The Computer and the Brain.
I think if von Neumann had lived, the history of artificial intelligence might
have been somewhat different. So, probabilities eventually found their
way into AI by something called graphical models,
Which are a marriage of graph theory and probability theory.
In the 1980's, there was a lot of work on expert systems in AI that use bags of
rules for tasks such as, medical diagnosis or exploring for minerals.
Now, these were practical problems so they had to deal with uncertainty.
They couldn't just use toy examples where everything was certain.
People in AI dislike probability so much that even when they were dealing with
uncertainty, they didn't want to use probabilities.
So, they made up their own ways of dealing with uncertainties that did not involve
probabilities. You can actually prove that this is a bad
bet. Graphical models were introduced by Pearl,
Heckman, Lauritz and many others who shared that probabilities actually worked
better than the ad hoc methods developed by people doing expert systems.
Discrete graphs were good for representing what variable dependent on what other
variables. But once you have those graphs, you then
needed to do real value computations that respected the rules of probability so that
you could compute the expected values of some nodes in the graph, given the
observed states of other nodes. Belief nets is the name that people in
graphical models give to a particular subset of graphs which are directed
acyclic graphs. And typically, they use sparsely connected
ones. And if those graphs are sparsely
connected, they have clever inference algorithms that can compute the
probabilities of unobserved nodes efficiently.
But, these clever of algorithms are exponential in the number of nodes that
influence each node, so they won't work for densely connected nodes.
So, belief net is directed acyclic graph composed of stochastic variables,
And here's a picture of one. In general, you might observe any of the
variables. I'm going to restrict myself to nets in
which you only observe the leaf nodes. So, we imagine is these unobserved hidden
causes, and they may be lead, And they eventually give rise to some
observed effects. Once we observe some variables, there's
two problems we'd like to solve. The first is what I call the inference
problem, and that's to infer the states of unobserved variables.
Of course, we can't infer them with certainty, so what we're after is the
probability distributions of unobserved variables.
And if unobserved variables are not independent of one another, given the
observed variables, there is probability distributions are likely to be big
cumbersome things with an exponential number of terms in.
The second problem is the learning problem.
That is, given a training set composed of observed vectors of states of all of the
leaf nodes, How do we adjust the interactions between
variables to make the network more likely to generate that training data?
So, adjusting the interactions would involve both deciding which node is
affected by which other node, And also deciding on the strength of that
effect. So, let me just say a little bit about the
relationship between graphical models and neural networks.
The early graphical models used experts to define the graph structure and also the
conditional probabilities. They would typically take a medical expert
and ask him how likely is this to cause that, and then they would make a graph in
which the nodes had meanings. And they typically had conditional
probability tables that described how a set of values for the parents of a node
would determine the distribution of values for the node.
Their graph was sparsely connected. And the initial problem they focused on
was how to do correct inference. Initially, they weren't interested in
learning because the knowledge came from the experts.
By contrast, for neural nets, learning was always a central issue and hand wiring the
knowledge was regarded as not cool. Although, of course, wiring in some basic
properties, as in convolutional nets, was a very sensible thing to do.
But basically, the knowledge in the net came from learning the training data, not
from experts. Neural networks didn't aim to have
interpretability or sparse connectivity to make the inference easy.
Nevertheless, there are neural network versions of belief nets.
So, if we think about how to make generative models out of idealized
neurons, There's basically two types of generative
model you can make. This energy based models, where you
connect binary stochastic neurons using symmetric connections, and then you get a
Boltzmann machine. A Boltzmann machine, as we've seen, is
hard to learn. But if we restrict the connectivity, then
it's easy to learn a restricted Boltzmann machine.
However, when we do that, we've only learned one hidden layer.
And so, we're giving up on a lot of the power of neural nets with multiple hidden
layers in order to make learning easy. The other kind of model you can make is a
causal model. That is a directed acyclic graph composed
of binary stochastic neurons. And when you do that, you get a sigmoid
belief net. In 1992, Neal introduced models like this
and compared them with Boltzmann machines and showed that Sigmoid belief nets were
slightly easier to learn. So, a Sigmoid belief net is just a belief
net in which all of the variables are binary stochastic neurons.
To generate data from this model, you take the neurons in the top layer.
You determine whether they should be ones or zeros based on their biases,
So you determine out stochastically. And then, given the states of the neurons
in the top layer, you'd make stochastic decisions about what the neurons in the
middle layer should be doing. And then, given their binary states, you
make decisions about what the visible effect should be.
And by doing that sequence of operations, a causal sequence from layer to layer,
You would get an unbiased sample of the kinds of vectors of visible values that
your neural network believes in. So, in a causal model, unlike a Boltzmann
machine, it's easy to generate samples.