Prediction, decision, control, lots of
techniques, many different ways of
stringing them together, deciding which
techniques to use sounds very complicated.
And, thinking about how to implement a
system like a global traffic management
system with lots of self driving cars
certainly is extremely complex.
But, there's actually a machine which is
doing many more complex things everyday
and there are about seven billion of them
on this planet already. Obviously, I'm
talking about the human brain.
Not only does it do all the tasks that we
have discussed, discussed in this course,
it does them with a fairly uniform
architecture.
So, the plasticity or uniform architecture
of the brain look something like this.
I mean, there are a whole bunch of
neurons,
About a hundred billion neurons in the
brain.
And, if you look at the top most layer of
the brain, a few millimeters thick,
It's called the, the neocortex.
Which is believed responsible for much of
our conscious thought.
The way it's organized is, is in bunches
of neurons which are arranged in columns,
so there's a vertical arrangement of
neurons.
And these neurons have connections
vertically as well as horizontally across
columns.
That's very important,
That's one element of that structure.
Each neuron, on the other hand,
Looks something like this.
The most of the white matter in one's
brain is consisting of the connections
between the neurons.
The neuron body is actually a small
fraction of that of the gray matter which
is there in the brain.
And the connections between neurons look
something like this.
These are called dendrites, and this is
the axon.
And, a dendrite connects to other neurons
via synapses.
Synapses we've learned in the past decade
form very rapidly between neurons that are
dendrites which are close to, dendrites
and axons which are close to each other,
and can, can, can form in minutes and can,
can decay over time, and very rapidly as
well. So, there's another important
feature of, of, of the brain.
An abstract model of the neuron which is
what is used by hierarchical temporal
memory, which is what we're going to talk
about, which is a model of the brain
profounded by Jeff Hawkins. and he talks
about this in a recent talk at the
International Symposium on, on Computer
Architecture, funny enough just in June
this year.
So, the model that he uses is an abstract
model of the neuron which looks lot like a
neural element in a neural network.
However, it has some very important
differences and we'll explain these, this
structure in the next few minutes as we go
along.
The first important feature of
hierarchical temporal memory is that it
relies on sparse representations.
In particular, sparse distributed
representations.
These are closely related to the sparse
distributed memory that we discussed way
back during the Locke lecture.
Remember, the properties of very long bit
sequences of zeros and ones.
In particular, if we have patterns of
thousand bits in length, we, we learned
that there is a very low chance that
patterns defer in less than 450 places.
So, most of them are far apart in this
sense, most random patterns chosen at
random are far apart.
Now, consider special types of patterns.
This time we'll take 2,000 bits because
that's the example that Jeff Hawkins uses
in his lecture.
So, we have pattern of 2,000 bits but it's
forced to have only 40 ones.
So, only two percent of these bits are
ones,
And we forced that in a particular way,
which we'll describe shortly.
But, if we do this, then lets see what
happens.
There's a very low chance of a random
sparse pattern, that means another random
pattern with only 40 ones, matching any of
these ones.
So, matching a significant number of these
40 ones.
Just imagine, if, if twenty of these ones
have to match then the chance is one by
2,000 multiplied by itself twenty times.
Even if we drop all the ten random
positions out of these 40, right?
Let's just, for example, we have a pattern
which has 40 ones,
But we decided to use, to retain only ten
of them at random.
And think about another sparse pattern of
40 ones to start with,
The chance that, after dropping ten, all
but ten from that pattern and all but ten
from our first pattern, the chance that
any of these ten match is again, very
small,
Right? Again, it's the one by 2,000
multiplied by itself many times and you
can work that out.
Please try to work it out because I, I
might even ask a question on this at some
point.
It's fairly simple arithmetic, not even as
complex as what we did for the zero one
sequences in sparse memory.
Now, this a, this particular feature is
exploited by hierarchical temporal memory
in the following way.
If this con, if you consider the bunch of
neurons in the sheet of neurons in the
brain, say, or models of the brain,
And say these neurons are being activated
by, by, by, by light intensity in the,,
following in a particular pattern, right?
Then, obviously, a light intensity pattern
won't just have two percent of it bright.
It will be many more pixels will be bright
in this kind of an image.
But, what this sparse representation does
is, it chooses the 40 brightest in the
same sense that it's not the 40 brightest
necessarily.
But, if, if a set of neurons is bright in
this area, they will be inhibited by
neurons which are bright nearby..
So, only one of these neurbons, neurons
out of these bunch which are bright, will
end up firing.
And by forcing neurons to turn off because
their neighborhood is also equally bright,
we get a sparse representation for a
fairly, otherwise fairly dense image.
Now, it's not clear why this is important
right now, but it'll become clearer, sort
of, in a few minutes.
What happens is that the similar scene,
Say that a similar scene this was a
particular pattern.
And you, you view a similar scene, it will
give this a very sparse pattern even after
sub sampling.
So, even after we get rid of 30 of the 40
ones, we'll still get a similar sparse
pattern from two different instances of
the same scene or even similar scenes.
Howeve, if you see completely different
scenes,
The chance that the sparse patterns that
we'll get match in a large number of
positions is very small.
And that's the very important part of
these kinds of representations,
That the key is the number of, of, of, of
base that we start with.
The fact that we choose only a small
number of them as ones, and randomly drop
some of them each time.
All these things put together make it very
unlikely that, that this similar scenes
will make the same sparse patterns.
But, similar scenes will very likely give
the same sparse patters, or sparse
patterns that match each other.