[MUSIC] So let's generate some data and
fit polynomials of increasing degrees and see what happens to
the estimated coefficients. So to start with, let's just import
some libraries that are gonna be useful. And then we're gonna just
create 30 different x values. So, in the end we're gonna create
a data set with 30 observations. And then what we're gonna do is, we're gonna compute the value
of this sign functions. So, evaluate the sign function
at these 30 x values. But of course when we're doing our
analysis we're going to assume we have noisy data so
we're gonna add noise to these true sign values to get
our actual observations. So here we're just adding noise. And then we're gonna put
this into an S frame. And so here's what our data looks like. We have a set of X values, and
our corresponding Y values, but of course, it's easier to just
visualize what this data set looks like. So let's just make a plot of X versus Y. So, here you can see that there's
an underlying trend like this so the true trend like we talked
about is this sin function. So it's going up and coming down here and
these black dots are our observed values. Okay, so now let's get to our
polynomial regression task and to start with what we're gonna
do is we're first just gonna. Define our polynomial features so what we're doing with this function
polynomial underscore features is we're taking our S frame and then we're just
gonna make a copy of that S frame. And for any degree polynomial
that we're considering, we're gonna manipulate the S frame to
include extra columns that are powers of X based on whatever degree we've
specified for that polynomial. So that's what this function does right
here and then the very important function is polynomial regression which is
implementing our multiple regression model using the features specified by this
polynomial underscore features function. So, again, for simplicity we're just
using graphlab create and we're gonna use the dot linear regression function
where the features we're specifying are just the powers specified by the
degree of the polynomial we're looking at. And then our target is our observation Y,
and then there are these two terms our l2 penalty and l1 penalty
that we set to be equal to zero. So this module on ridge regression is
gonna be all about this l2 penalty, and we're gonna get to that. And then the next module is gonna
be all about this l1 penalty, but for now, let's just understand
that if we set these values to zero we just return to our
standard least squares regression. Okay, so that's what our polynomial
regression function is doing. And the next function we're gonna
define allows us to plot our fit. And finally we're gonna define
a function that allows us to, in a very nice way, print the coefficients
of our polynomial regression. And for this we're gonna use this
numpy library because that allows for this really pretty printing
of our polynomial. Okay, so now we're gonna use
all these functions again and again as we explore different degrees
of polynomials fit to this data. So to start with let's just
consider fitting a very low order degree two polynomial. So first we're gonna do our polynomial
regression fit taking our s-frame, which we call data, and
specifying that the degree is two for this polynomial regression. Then let's look at the coefficients
that we've estimated. And here's where we've done this really
nice printing of these coefficients using that NumPy library where
here what we see is that we have some coefficient on X squared,
a coefficient on X, and just our intercept term here. And these values are, I don't know
how to call them reasonable or not reasonable, but
they're relatively small numbers. Number we can kind of appreciate,
number like five and four and something close to zero. And now let's plot what our
estimated fit looks like. And this looks pretty good. It's a nice smooth curve. Goes between the values pretty well, and in between values you'd imagine
believing what this fit is. But now let's go to a slightly
higher degree polynomial just a order four polynomial. And here we're doing all the steps at
once, where we're going to fit our model, print the coefficients and plot the fit. And so if we look at
the estimated coefficients of our fourth order polynomial, we see that the
coefficients have increased in magnitude. We have numbers like 23 and 53 and a 35. And the fit is looking a bit wigglier. Still actually looks pretty reasonable but
now let's get to our degree 16 polynomial. So remember we only have
30 observations and we're trying to fit 16 order polynomial. So what happens here? So in this case, we see that the coefficients have
just become really, really massive. Here we have 2.583 times 10 to the 6 and
here 1.295 times 10 to the 7th. So these are really, really,
really large numbers. And let's look at the fit. As expected,
this fit is also really wiggly and crazy. And we probably don't believe that this
is what's really going on in this data. So this is an example pictorially
of an overfit function. But what we see in the take-home message
from this demo is the fact that when we're in these situations of being very overfit,
then we get these very, very, very large estimated coefficients
associated with our model. So yeah, whoa,
these coefficients are crazy. So what ridge regression is gonna
do is it's going to quantify overfitting through this measure of
the magnitude of the coefficients. [MUSIC]