Whatever our model,
One needs to minimize f of x minus y.
Otherwise, the, the data, the value that
you want to predict you want to minimize
the error between the, what you already
have.
Real values and your predicted values for
those real values..
Well, f is complicated, there's no formula
like the normal equations that we had for
linear least squares.
So, all these methods, whether they are
regression with logistic function, support
vector machines, neural networks, all
learn the parameters through uniterative
process.
We start with some f which is a
parameterization of the function f,
It maybe not be linear like f transposed
x.
It could be some other parameterization
so, you know, you could have squares, you
could have exponentials, you can all kinds
of things.
But, you have some parameters which tells
you what the function f for each choice of
parameters is.
And, we need to refine it alliteratively
to find the f which minimizes the error.
In the world of neural networks, the first
search algorith was called back
propagation which has nothing going to do
with the feed back, neural networks, is
just a new algorithms, it is just going to
learn self.
While the other areas which were probably
more mathematically founded, straight
forward optimization techniques were used
to find the minimum of this error.
So, if we start off with a guess and we
choose maybe a slightly different f as two
different guesses,
One way to start minimizing this error or
moving towards a value of f which is
likely to have lower error, is to use
something called gradient descent.
This formula looks a bit daunting so let
me explain it in simple language.
Epsilon of f is a function of the
parameters of f.
So, one can measure how fast this function
changes if one changes each component of f
separately, which is nothing but taking
the partial derivative of the function
with respect to that component.
String all those partial derivatives
together, and you'll get a vector which is
just this del sub f of epsilon.
That vector tells you which direction to
move each component of f so as to reduce
subset.
So, for example, the first component of f
is such that if you increase it, then
epsilon decreases, then you would choose
to increase that component of f.
If the second component is one which
decreases epsilon, only if you decrease
that component, you will decrease f so
then the, that particular element of del
would be negative.
So, you'd get this vector del, computed
using your own value of f of r of f.
That means, your previous iterations start
with f0, for example. And then, add some
small multiple of this derivative to your
current guess to create a new guess.
Now, you need to compute these derivatives
so you use the difference is between
epsilon at f of i and eps, epsilon at f of
i minus one divided by, you know, f of i
minus f of i minus one component-wise to
get estimates for this derivative and
that's why you needed two guesses to start
with so, because you have to bootstrap
this duration.
Now, this is a very popular and very
simple method and you can have.
More complicated ways of finding the
minimum value of, of epsilon by, by using
Newton's iteration where you use second
derivatives in addition to first
derivatives and various different
techniques in the mathematics literature
are available to you.
This works fine if you have numbers, that
means if you have the x's are actual
values as we've been assuming, but there
are, of course, some caveats.
This iteration sort of works and all
situations work provided there is only,
you know, one minima and you're actually
starting very close to that minima if, if
there are multiple minimas.
And if there are many different minimas,
you might get stuck in a local minima, and
not, and not really get to the real
minimum value.
So, so you need better techniques.
The other thing would be that the function
f might not necessarily be parametrized by
a single set of values there may be some
constraints.
You might have a function that says, that
you have five different lines which meet
at, at the, the connecting points and
that's your function.
So, you're going to try to fit five lines
to all your data.
Then, you have constraints which will then
make this situation a little bit more
complicated and, and, and, and, complex as
well.
But, we won't go into those details right
now.
The other problem is, is if x is not
numbers and if x is categorical, at least
some of the x are categories like yes, no,
red, blue, or green, then you somehow have
to convert them to a binary form.
So, if there are n categorical variables,
you might want to convert, convert them
to, for example, zero or one.
Of course, now, you're going to have many
more than n.
So, you would say like words.
You, you have categorical variables which
are words and you suddenly increase the
dimension of, you know, space from say,
You're, you're talking about one word, but
words could be out of a set of millions.
So now, you have a million dimensional
space and it's zero or one depending on
whether that word is there.
And that tremendously complicates such
methods and they often become unviable if
your categorical data is highly, very high
dimension and you have to use different
techniques to learn parameters.
You could also convert the these
techniques, these categorical variables to
real numbers through a positive, through a
process of reversed classification, which
we wont talk about, but essentially there,
you, you think about high, low, and medium
as function which are not fixed as being
high, or low, or medium but could vary as
something being 50% high, twenty percent
low, etc.
And then, use such fuzzy membership to
convert your categorical variables to, to,
into, into real numbers.
And then, once you have real numbers, you
can use this technique or you could deal
with the space of categorical data itself.
And then, there are techniques like
neighborhood search, heuristic AI search,
genetic algorithms, and such techniques
which don't rely on converting your
parameters into real space.
And then, you get the problem of a
combinatorial explosion and you have to
use all kinds of techniques to figure out
good ways of minimizing the error.
And lastly, as we have seen in previous
lectures, probabilistic models can deal
with probabilities of variables taking on
particular categorical values as opposed
to having to convert those to numbers or
deal with them directly.
And these techniques have become
increasingly powerful and, and popular,
precisely because you're able to deal
with, with the numbers without having to
get into too much exponential
combinatorial blowup.
Now, that's quite a mouthful and many of
you might not have got it.
But those of you who are on the verge of
graduate research, might want to listen to
that spill a couple of times cuz it does,
sort of, unify different or sort of give a
bigger picture of how learning parameters
in predictive models work and what are the
possible options.
There are some more linkages which I'd
like to explore.
For example,
Here we are trying to minimize the error
in a predictive model.
In other situations, we might need to
maximize a choice which is want to find
the best way to transport goods across the
country,
Or the best way to allocate one's
advertising funds across different
channels.
So, there you're trying to maximize some
function and that's related closely to the
problem of minimizing a prediction error.
Similar techniques, gradient-descent,
Newton's method and various others apply
all the similar, similar problems in terms
of categorical data,
Local minima, and constraints all apply as
well.
Then, once one has decided where one wants
to go, one needs to figure out how to get
there.
So typically, if you're driving a car, for
example, you have a system and your angle
of steering could be, say, theta.
And every time you, you, you change your
theta, your car will move, move in a
certain direction so you'll have a, a new
state of your system.
And the goal is to figure out which
sequence of control actions will
ultimately minimize the difference between
the state of your system and where you
want that state to be.
And there could be other constraints that
you want that paths, paths to be fairly
smooth, etc.
So, you get all those problems of
constraints again.
Again, it's a minimization problem.
And here, unlike in the first two cases of
minimizing the prediction error and, or
optimizing a function each iteration is
actually executed and the response of a
system, rather than a function kind of
abstractly is what you're dealing with.
So, an actual control system will make a
change,
See what the output is and make another
change again, techniques like
gradient-descent,, Newton's method, and
all that start applying, but now you're
controlling actions.
So, from predicting the future or
predicting where things are going to go to
figuring out how to best utilize that
information to optimize one's own goals.
And then, to decide how to change ones
behavior or change ones system, so as to
get to the gold state, all these are
tightly related.
And, I hope you've seen that the
mathematics is related, now lets talk
about some of the applications.
Before we do that, of course, in practice,
whether you're dealing with support vector
machines, logistic regression, neural
networks, or what have you, you won't ever
really have to do these things.
There are packages available that let you
choose let, let you apply any of these
techniques without having to get into the
math, or get into the duration.
The important thing is, of course, trying
to choose which package to use. And, as
soon as we do the applications,
I'll try to summarize which package to
use, what kind of techniques to use from
my experience.
And then, we move on to more issues of
deep belief networks.