We're now going to talk a little bit about an issue that's of interest according to
scientists, but may not be of much interest to engineers.
So if you're an engineer you can just ignore this video.
In computer science, there's been a debate going on for maybe a 100 years about the
relationship between feature vector representations of concepts and
representations of concepts by their relations to other concepts.
And the learning algorithm we've just seen for family trees has a lot to say about
that debate. We're now going to make a brief diversion
into cognitive science. There's been a long debate between two
rival theories of what it means to have a concept.
The feature says a concept is a big set of semantic features.
This is good for explaining similarities between concepts.
And it's convenient for things like machine learning.
Because we like to deal with vectors of activities.
The structuralist theory says that the meaning of a concept lies in its
relationships to other concepts. So conceptual knowledge is best expressed
not as a big vector, but as a relational graph.
In the early 1970s, Marvin Minsky use the limitations of perceptrons as evidence
against featured actors in favor of relational graph representations.
My belief is that both sides in this debate are wrong because both sides
believe that the two theories are rivals and they're not rivals at all.
A neural net can use vectors of semantic features to implement a relational graph.
In the neural network that learns family trees, we can think of explicit inference
as I give you person one and I give you a relationship then you tell me person two.
And to arrive at that conclusion, the neural net doesn't follow a whole bunch of
rules of inference. It just passes information forward through
the net. As far as the neural net is concerned, the
answer is intuitively obvious. Now if you look at the details of what's
happening, there's lots of probabilistic features that are influencing each other.
We call these microfeatures to sort of emphasize they're not like explicit
conscious features. In a real brain, there might be millions
of them and millions of interactions, and as a result of all these interactions, we
can make one step of explicit inference. And that's what we believe is involved in
just seeing the answer to something. There are no intervening conscious steps,
but nevertheless there's a lot of computation going on in the interactions
of neurons. So we may use explicit rules for conscious
deliberate reasoning, but a lot of our common sense reasoning, particularly
analogical reasoning, works by just seeing the answer, with no conscious intervening
steps. And even when we do conscious reasoning,
we have to have some way of just seeing which rules apply, in order to avoid an
infinite regress. So, many people, when they think about
implementing a relational graph in a neural net, just assume that you should
make a neuron correspond to a node in the relational graph and a connection between,
two neurons correspond to a binary relationship.
But this method doesn't work. For a start, the relationships come in
different flavors. They're are different kinds of
relationship. Like mother of, or aunt of and, a
connection in a neural net only has a strength.
It doesn't come in different types. Also we need to do we turn around your
relationships like'a' is between'b' and'c'.
We still don't know for sure the right way to implement relational knowledge in a
neural net. But it seems very probable that many
neurons are used for representing each of the concepts we know, and each of those
neurons is probably involved in dealing with many different concepts.
This is called a distributed representation.
It's a many to many mapping between concepts and neurons.