Sometimes people talk about support
vector machines, as large margin
classifiers, in this video I'd
like to tell you what that
means, and this will
also give us a useful
picture of what an SVM
hypothesis may look like.
Here's my cost function for the support vector machine
where here on the left
I've plotted my cost 1
of z function that I used for positive examples and on the right I've  plotted my
zero of 'Z' function, where I have
'Z' here on the horizontal axis.
Now, let's think about what
it takes to make these cost functions small.
If you have a positive example,
so if y is equal to
1, then cost 1 of
Z is zero only when
Z is greater than or equal to 1.
So in other words, if you
have a positive example, we really
want theta transpose x to be greater
than or equal to 1
and conversely if y is
equal to zero, look this
cost zero of z function,
then it's only in
this region where z is
less than equal to 1
we have the cost is zero
as z is equals to zero,
and this is an interesting property of the support
vector machine right, which is
that, if you have a positive
example so if y is equal to one,
then all we really need
is that theta transpose x is greater than equal to zero.
And that would mean that we classify correctly
because if theta transpose x is greater than zero our
hypothesis will predict zero.
And similarly, if you have
a negative example, then really all you want is that theta transpose x is
less than zero and that will make sure we got the example right.
But the support vector machine wants a bit more than that.
It says, you know, don't just barely get the example right.
So then don't just
have it just a little bit bigger than zero. What
i really want is for this to be
quite a lot bigger than zero
say maybe
bit greater or equal to one
and I want this to be much less than zero.
Maybe I want it less than or
equal to -1.
And so this builds in an
extra safety factor or safety
margin factor into the support vector machine.
Logistic regression
does something similar too of course,
but let's see what
happens or let's see what
the consequences of this are, in the
context of the support vector machine.
Concretely, what I'd like to do next is
consider a case
case where we set
this constant C to be
a very large value, so let's
imagine we set C to
a very large value, may be a hundred thousand, some huge number.
Let's see what the support vector machine will do.
If C is very,
very large, then when minimizing
this optimization objective, we're going
to be highly motivated to choose
a value, so that this
first term is equal to zero.
So let's try to
understand the optimization problem in
the context of, what would
it take to make this
first term in the objective
equal to zero, because you
know, maybe we'll set C to
some huge constant, and this
will hope, this should give us
additional intuition about what
sort of hypotheses a support vector machine learns.
So we saw already that
whenever you have a training
example with a label
of y=1 if you
want to make that first term
zero, what you need is
is to find a value of theta
so that theta transpose x i is greater
than or equal to 1.
And similarly, whenever we have an example,
with label zero, in order
to make sure that the cost,
cost zero of Z,  in order to
make sure that cost is
zero we need that theta transpose x i
is less than or
equal to -1.
So, if we think
of our optimization problem as
now, really choosing parameters
and show that this first
term is equal to zero,
what we're left with is
the following optimization problem.
We're going to minimize that first
term zero, so C
times zero, because we're going
to choose parameters so that's equal
to zero, plus one half
and then you know that
second term and this
first term is 'C' times zero,
so let's just cross that
out because I know that's going to be zero.
And this will be subject to the constraint
that theta transpose x(i)
is greater than or equal to
one, if y(i)
Is equal to one and
theta transpose x(i) is less than
or equal to minus one
whenever you have
a negative example and it
turns out that when you
solve this optimization problem, when you
minimize this as a function of the parameters theta
you get a very interesting decision
boundary. Concretely, if you
look at a data set
like this with positive and negative examples,  this data
is linearly separable and by
that, I mean that there exists, you know, a straight line,
altough there is many a different straight lines,
they can separate the positive and
negative examples perfectly.
For example, here is one decision boundary
that separates the positive and
negative examples, but somehow that
doesn't look like a very
natural one, right? Or by
drawing an even worse one, you know
here's another decision boundary that
separates the positive and negative examples
but just barely.
But neither of those seem like particularly good choices.
The Support Vector Machines will instead choose this
decision boundary, which I'm drawing in black.
And that seems like a much better decision boundary
than either of
the ones that I drew in magenta or in green.
The black line seems like a more
robust separator, it does
a better job of separating the positive and negative examples.
And mathematically, what that does is,
this black decision boundary has a larger distance.
That distance is called the margin, when I
draw up this two extra
blue lines, we see
that the black decision boundary has
some larger minimum distance from any of my training examples,
whereas the magenta and the green lines
they come awfully close to the training examples.
and then that seems to do a less a good job separating
the positive and negative classes than my black line.
And so
this distance is called
the margin of the
support vector machine and this
gives the SVM a certain
robustness, because it tries
to separate the data with as
a large a margin as possible.
So the support vector machine is
sometimes also called a large
margin classifier and this
is actually a consequence of
the optimization problem we wrote down on the previous slide.
I know that you might be
wondering how is it that
the optimization problem I wrote
down in the previous slide, how
does that lead to this large margin classifier.
I know I haven't explained that yet.
And in the next video
I'm going to sketch a
little bit of the intuition about why
that optimization problem gives us
this large margin classifier. But
this is a useful feature to
keep in mind if you are
trying to understand what are the
sorts of hypothesis that an SVM will choose.
That is, trying to separate the
positive and negative examples with as big a margin as possible.
I want to say one last thing
about large margin classifiers in
this intuition, so we
wrote out this large margin classification
setting in the case
of when C, that regularization concept,
was very large, I think
I set that to a hundred thousand or something.
So given a dataset
like this, maybe we'll choose
that decision boundary that
separate the positive and negative examples on large margin.
Now, the SVM is actually sligthly
more sophisticated than this large
margin view might suggest.
And in particular, if all you're
doing is use a large
margin classifier then your
learning algorithms can be sensitive
to outliers, so lets just
add an extra positive example
like that shown on the screen.
If he had one example then
it seems as if to separate
data with a large margin,
maybe I'll end up learning
a decision boundary like that, right?
that is the magenta line and
it's really not clear that based
on the single outlier based on
a single example and it's
really not clear that it's
actually a good idea to change
my decision boundary from the black
one over to the magenta one.
So, if C, if
the regularization parameter C were very
large, then this is
actually what SVM will do, it will
change the decision boundary
from the black to the
magenta one but if
C were reasonably small if
you were to use the C,
not too large then you
still end up with this
black decision boundary.
And of course if the data were not linearly separable so if you had some positive
examples in here, or if
you had some negative examples
in here then the SVM
will also do the right thing.
And so this picture of
a large margin classifier that's
really, that's really the
picture that gives better intuition
only for the case of when the
regulations parameter C is
very large, and just
to remind you this corresponds
C plays a role similar to
one over Lambda, where Lambda
is the regularization parameter
we had previously. And so it's
only of one over Lambda
is very large or equivalently
if Lambda is very small that
you end up with things like
this Magenta decision boundary, but
in practice when applying support vector machines,
when C is not very very large
like that,
it can do a better job ignoring
the few outliers like here. And
also do fine and  do reasonable things
even if your data is not linearly separable.
But when we talk about bias and variance in the context of support vector machines
which will do
a little bit later, hopefully all
of of this trade-offs involving the regularization
parameter will become clearer at
that time. So I hope
that gives some intuition about
how this support vector machine functions as
a large margin classifier that
tries to separate the data with
a large margin, technically this
picture of this view is true
only when the parameter C is very large, which is
a useful way to think about support vector machines.
There was one missing step in
this video which is, why is
it that the optimization problem we
wrote down on these
slides, how does that actually
lead to the large margin classifier, I
didn't do that in this video,
in the next video I
will sketch a little bit
more of the math behind that
to explain
that separate reasoning of how
the optimization problem we wrote out
results in a large margin classifier.