In this video, I'll talk about how we can control compacity by limiting the size of
the weights. The standard way to do this is to
introduce a penalty that prevents the weights from getting too large.
With the implicite assumption that a network with small weights is somehow
simpler than a network with large weights. There are several different penalty terms
we can use. And it's also possible to constrain the
weights, so that the incoming weight vector to each of the hidden units is not
allowed to be longer than a certain length.
The standard weight can limit the size of the weights, is to use an L2 weight
penalty, which means that we penalize the squared value of the weight.
This is sometimes called weight decay in the neural network literature, because the
derivative of that penalty acts like a force pulling the weight towards zero.
This weight penalty keeps the weight small, unless they have big urge
derivatives to counteract it. So if you look at what the penalty term
looks like, as the weight moves away from zero, you get this parabolic cost.
If you look at the equation, the cost that you're optimizing is the normal error that
you're trying to reduce, plus a term which is the sum of the squares of the weights,
with a coefficient in front, Lambda. And we divide by two so that when we
differentiate the two is cancelled. That coefficient in front of the sum
squared weights is sometimes called the weight cost.
That determines how strong the penalty is. If you differentiate, you can see the
derivative of this cost is just the derivative of the error plus something
that's to do with the size of the weight and the value of Lambda.
That derivative will be zero when the magnitude of the weight is one over Lambda
times the magnitude of the derivative. So the only way you can have big weights,
when you're at a minimum of the cost function, is if they also have big error
derivatives. And this makes the weights much easier to
interpret. You don't have a whole lot of weights that
are large but aren't doing anything. The effect of an L2 penalty on the weights
is to prevent the network from using weights that it doesn't need.
This often improves generalization a lot, because it can use those weights that it
doesn't really need to fit the sampling error.
It also makes a smoother model in which the output changes more slowly as the
input changes. So if the network has two very similar
inputs, when you put in an L2 weight penalty, it prefers to put half the weight
on each of those two similar inputs rather than all of the weight on one, as shown on
the right here. If the two inputs are very similar, those
two networks will produce very similar outputs.
But the one with the half weights will have much less extreme changes in its
outputs when you change the inputs. There are other kinds of weight penalty.
For example, an L1 penalty, where the cost function is just this v shape.
So here what we're doing is we're penalizing the absolute values of the
weights. This has the nice effect that it drives
many of the weights to be exactly zero and that helps a lot in interpretation.
If there's only a few non zero weights left, it's much easier to understand
what's going on. We can also use weight penalties that are
more extreme than L1 where the gradient of the cost function actually gets smaller
when the weight gets really big. This allows the network to keep large
weights without them being pulled towards zero.
It's just the small weights that'll get pulled towards zero.
So we then even more like that getting it with a few large weight.
Instead of putting penalties on the weights, we could actually use weight
constraints. What I mean by that is instead of
penalizing the squared value of each weight separately, we put a constraint on
the maximum squared length of the incoming weight vector of each hidden unit or
output unit. When we update the weights, if the length
of that incoming vector gets longer than allowed by the constraint, we simply scale
the vector down by dividing all the weights by the same amount until its
length fits the allowed length. Using weight constraints like this, has a
number of advantages over weight penalties, and I found these work
necessary better. It's much easier to select the sensible
value for the squared length of the incoming weight factor than it is to
select the weight penalty. That's because, logistic units.
Have, a natural scale to them so we know what a weight of one means.
Using weight constraints also prevents hidden units getting stuck near zero with
all their weights being tiny and not doing anything useful.
Because when all their weights are tiny, there's no constraint on the weights.
So there's nothing preventing them growing.
Weight constraints also prevent the weight from exploding.
One of the subtle things that weight constraints do is that when a unit hits
its constraint, the effective penalty on all of its weights is determined by the
big gradients. So if some of the incoming weights have
very big gradients, they'll be trying to push the length of the incoming weight
factor up. And that will push down on all the other
weights. So in effect, if you think of it like a
penalty, the penalty scales itself so as to be appropriate for the big weights and
to suppress the small weights. This is much more effective than a fixed
penalty of pushing irrelevant weights towards zero.
For those of you who knows about La Grange multipliers,
The penalties of in just the La Grange multipliers required to keep the
constraints satisfied.