So now, we'll go beyond least squares.
However, let me mention that the technique
that we're going to cover this week is
least squares, which have already been
covered. And, what we will follow is
essentially a broad overview of a variety
of different areas dealing with different
types of prediction models and their
relationships which are increasingly
becoming apparent in research community.
Let's first see what happens if we have
categorical data. That is that, the target
variables that we are trying to predict,
the y's, are not numbers but yes is a
node, which is essentially the learning
problem that we had earlier.
Well, the challenge here is that the x's
are still numbers. So, it might not be
such a great idea to use something like a
naive Bayes classifier because one would
have to discretize those numbers into
categorical variables which might not be
appropriate.
So, one tends to try to use regression
techniques. But, using a linear regression
when all one is trying to do is separate
out the no's which is the reds from the
blues, which may be yes's.
It's quite brittle because the points
right near the line might get classified
either way, and the distinctions one has
to make are very small and are subject to
a lot of error.
The other problem, of course, is that
there may be no such lines separating the
data and we'll come to that in a minute.
But, just to make regression work for
categorical data, what the most common
technique being used today is logistic
regression. Which essentially replaces f
by a function of the form one - one over e
to the -f transposed x, instead of f
transposed x.
Let's see how this function behaves.
Now, suppose f transposed x is a very
large value.
Then, even the -f transposed x is close to
zero,
Which means that one up, one over that is
close to infinity and we have one minus of
that, becomes a very,
As f transposed x is close to zero, F of x
is close to zero because you have one over
one,
And one minus of that is almost zero.
But if, as f transposed x increases,
The denominator here rapidly decreases to
almost zero which makes the second term
here rapidly increased to infinity, where
f transposed x very rapidly goes away from
zero to a negative value.
On the other hand, if f transposed x is,
becomes a negative value,
Even the x transpose x rapidly increases
towards infinity making the second term
almost zero. So, fx) of x goes close to
one.
So, the speed at which the second term
deviates from one is what makes logistic
regression particularly attractive to
separate out the yes's from the no's.
Of course, trying to fit such a function
is not as easier as linear squares.
But I'll show how that can be done very
soon.
The other problem with linear regression
is that the data might not be separable
using a line.
So, one might, for example, you might have
a parabolic relationship, as we described
earlier.
To support more complex functions, and
when one doesn't even know what kind of
function to have,
The technique called support vector
machines have become very popular. And
they're the kind of function, f, or is
parameterized by kernel parameters, and
these parameters are also learned along
with the, the function itself.
So, not only does the we learn the
function, but we also learn the type of
function from the data.
And so, if it's a parabola, you learn a
parabola.
If it's something more complicated, like a
circle with a hole inside, rather like a
doughnut, the support vector machine will
learn such functions as well.
The point I'm trying to make is that
starting from linear least squares, one,
one increases the complexity of the
functions one is trying to learn.
And we arrive at other types of
regressions and support vector machines.
Way back in the early days of the AI, one
found many efforts to mimic the brain,
which essentially resulted in the field of
neural networks.
Where one tried to create structures that
sort of look like how neurons in the brain
affect each other, based on their
connections to other neurons.
The first neural networks created in the
50's by McCullouch, Pitts, Rosenblatt and
others essentially, look like linear
combinations of inputs.
So, they were pretty much like least
squares, in the sense that we had, you
would have a neural network, which would
say that these are neurons and the input
to this neuron is the rainfall and this
one's the temperature. This one's the
harvest rainfall, and then we have another
one which has just one.
And, the output would be the wine quality
and the weights, that is the connections
between these neurons would be learned by,
by a process of iterative refinement which
could be essentially looked upon as a way
of solving the least squares problem.
To deal with exactly the same problems of
classification, we found logistic
functions being included in neural
networks.
So essentially, the, the, this field was
evolving parallely to statistical
prediction techniques.
And interest in neural networks, sort of
wind over the years along the way, of
course, we figured out different types of
neural networks which were multi-layer.
And essentially, by combination of
logistic functions, least linear
functions, and other types of functions,
one could learn more complex non-linear
functions of the input variables.
Hidden layers were introduced which were
essentially creating more complicated
forms of f.,
And parametrized by the weights which
would then get learned through a process
of optimization as we'll see shortly.
Finally, in recent years, in spite of the
fact that interest in neural networks has
waned over the years, these have become
interesting once again with the notion of
feedback.
Notice that these networks that are shown
here,
All the links go forward from the input
nodes to the output nodes.
So, they are feed forward even though
they're multi-layer.
But, when we start having feedback,
That is that an input, a node in the
middle, output is fed back into the
previous nodes this starts behaving like a
Bayesian belief network.
In particular, feedback neural networks
display the same kind of explaining away
effect which we found in Bayesian networks
that once you figure out that something, a
particular cause is, is more likely, other
causes which might also contribute to this
input node become less likely.
So, that makes this much more interesting
than the older neural networks and has
sparked much recent interest in this
field.
The field of deep belief networks,
Multilayer-feedback neural networks,
Temporal neural networks,
All this are all coming together and deep
relationships between them are being
explored.
We'll, we'll come to these, these points
soon.
But, for the moment, lets try to
understand,
If one has such a complex f, how would one
learn the parameters of f?