In this video, I'll talk about another way of restricting the capacity of a neural
network. We can do that by adding noise, either to
the weights or to the activities. I'll start by showing, that if we add
noise to the inputs in a simple linear network, that's trying to minimize the
squared error, that's exactly equivalent to imposing an L2 penalty on the weights
of the network. I'll then describe uses of noisy weights
in more complicated networks and I'll finish by describing a recent discovery
that extreme noise in the activities can also be a very good regularizer.
So let's look at what happens if we add Gaussian noise to the inputs to a simple
neural network. The variance of the noise gets amplified
by the squared weights on the connections going into the next hidden layer.
If we have a very simple net, with just a linear output unit that's directly
connected to the inputs, the amplified noise then gets added to the output.
So if you look at the diagram, We put in an input Xi with additional
Gaussian noise that's sampled from a Gaussian with zero mean and variant sigma
I^.. That additional noise has it's variants
multiplied by the squared weight. It then goes through the linear output
unit j. And so what comes out of j is the yj that would have come out before plus
Gaussian noise but has zero mean and has variance Wi^ sigma I^..
This additional variance makes an additive contribution to the squared error.
You can think of it like Pythagoras theorem, that the squared error is going
to be the sum of the squared error caused by Yj, and this additional noise, because
the noise is independent of Yj. So when we minimize the total squared
error, we'll minimize the squared error that will come out if it was a noise-free
system. And in addition, we'll be minimizing that second term.
That is, we'd be minimizing the expected squared value of that second term and the
expected squared value is just Wi^2, sigma I^2, so that corresponds to an I2 penalty
on wi with a penalty strength of sigma I^2.
For those of you who like math, I'm gonna derive that on this slide.
If you don't like math, you can just skip this slide.
The output, Y noisy, when we add noise to all of the inputs, is just what the output
would have been with noise-free system. The sum of all the inputs at wixi, plus wi
times the noise that we added to each input.
And those noises are sampled from a Gaussian with zero mean of variance sigma
I^..so So if we compute the expected squared
difference between Y noise E and the target value t, that's the quantity that's
shown on the left-hand side of the equation.
And I'm using an e followed by square brackets to mean an expectation.
That's not the arrow, that's an expectation.
And what we're computing the expectation of is the thing in the square brackets.
So in this case, we're computing the expectation of the squared arrow that
we'll get with the noisy system. So if we substitute the equation above for
Y noisy, we need the expectation of Y. Plus the sum of all the I, WI, epsilon I
So when we complete the square, the first time we get is Yt^ and that's not in the
side of expectation bracket because it doesn't involve any noise.
The second term is the cross product of the two terms above and the third term is
the square of the last term. Now that equation simplifies a lot.
In fact, it simplifies down to the normal squared error.
Plus the expectation of WI^2, epsilon I^2, summed over all I.
The reason it simplifies is because epsilon I is independent of epsilon J.
So if you look at the last term, when we multiply at that square, all of the cross
terms have an expected value of zero. Because we're multiplying together two
independent things that are zero mean. If you look at the middle chart, that also
has an expectation of zero, because each of the epsilon I's is independent of the
residual error. So we can rewrite the expectation of the
sum over all I of Wi^ epsilon squared, as simply the sum over all I with w I
squared, sigma I squared, because the expectation of up to I squared is just
sigma I squared, because that's how we generated epsilon i.
And so we see that the expected squared error we get is just the squared error we
get in the noise free system. Plus this additional term.
And that looks just like an L2 penalty on the WI.
With the sigma I^ being the strength of the penalty.
In more complex nets, we can restrict the capacity by adding Gaussian noise to the
weights. This isn't exactly equal to an L2 penalty.
But it seems actually to work better, especially in recurring networks.
So Alex Graves recently took his recurrent net that recognizes handwriting and tried
it with noise added to the weights. And it actually works better.
We can also use noise in the activities as a regularizer So suppose we use back
propagation to train a multi-lanural match with logistic hidden units.
What's gonna happen if we make the units binary and stochastic on the forward pass
but then we do the backward pass as if we'd done the normal deterministic forward
pass using the real values? So we're going to treat a logistic unit,
in the forward pass, as if it's a stacastic binary neuron.
That is, we compute the output of the logistic P, and then we treat that P as
the probability of outputting a one. And in the forward pass, you make a random
decision whether to output a one or a zero using that probability.
But in the backward paths, you use the real value of p for back propagating
derivatives through the hidden unit. This isn't exactly correct, but it's close
to being a correct thing to do for the stochastic system if all of the units make
small contributions to each unit in the layer above.
When we do this the performance on the training set is worse and training is
considerably slower. It may be several times slower.
But it does significantly better on the test set.
This is currently an unpublished result.