[MUSIC]
At the core of boosting is the idea of an ensemble classifier.
So let's revisit the idea of, say, decision stump.
A decision stump may take our data as input and might look at some particular
feature, let's say, income and ask, is income greater than 100K?
If yes, we say a loan is safe, and if not, we might say the loan is risky.
And so the output here, y hat, is +1 or
-1, depending on whether we think the loan is safe or risky.
We're going to use the following notation just to be clear in today's module.
We're going to use f(x) to denote one of these weak simple classifiers or
a classifier in general, so that's the f.
And so what boosting does starts not just from a single decision stump,
but it combines many of them together in order to improve the prediction.
So, for example, you may have a decision stump that splits on income,
another one that splits on credit history, another one on how much savings you have,
and the market conditions, and so on.
So if you're giving a particular input, for example, your income is 120K,
so the first decision boost here is going to vote with f1,
that the loan is safe, so it's going to vote +1.
But your credit history is bad, so
the second decision stump here is going to vote -1, f2.
And, say, the third one, because your savings are low,
is going to vote risky -1, and the third one,
market conditions, turned out to be good so you're going to vote +1.
And the question here the boosting asks is, how do you combine in an ensemble
way this -1, +1 votes from different classifiers, f1 through f4, in this case.
And so the way that we combine is by having some sort of weights that deal with
what is called the ensemble models.
So the weights here are w1, w2, w3, w4.
And we multiply the outputs from each of the classifiers, add them up together and
then take the sign of that.
If the sign is positive, we output +1.
If the sign is negative, we output -1.
So it's kind of a weighted voting schema.
This weighted voting schema, amazing, really awesome.
And let's see a little bit more of what that looks like.
So let's suppose that we have a particular set of weights and
we have multiple decision stumps, so classifiers that have provided their vote.
Let's see more explicitly what that output value would be.
So, for example, if the weights of each classifier is shown on the right here,
then we're going to say that output that we're going to predict is going to be
the sign of w1, which is 2.
Then multiply the vote of f1, which is +1 plus w2, which is 1.5, so
we believe that the second decision stump is not as important as the first one,
and the vote there was -1, so that multiplies -1.
Then, for the third one, again, we say that the weight is 1.5, and
again we say it's not as important as the first one, but
as important as the second one, and the prediction there was -1.
And finally, for the last one, we might say that the weight is 0.5.
So this is the least important of all of these classifiers.
And the function here that, the output of f4 here was +1, was a safe loan.
So we have these votes, these weights, we multiply them together and
we see that the output of this F classifier for
the input xi is the sign of -0.5, which is equal to or
implies that y hat i is -1.
So even though there was two positive votes and two negative votes, and the most
important classifier, the first one, was a positive vote, when you add it up and
you average the results, you get a risky loan as an output.
Now this is a simple example of what's called an ensemble classifier or
the combination multiple classifiers.
As we'll see, this kind of ensemble model,
this kind of combination is what everybody uses in industry to be able to
solve complex decision problems, complex classification problems.
Just to make sure we're all on the same page,
I'm going to just formally define the ensemble learning problem.
So we're given some data, where y is either +1 or -1,
there's also multiclass version of this, we're just going to talk about +1 or
-1 in today's module, and we have some input, x.
We have some data that allows us to learn f1, f2, all the way to fT,
which are the T weight classifiers, or just classifiers, that we learn from data.
And then some coefficients that we learn from data, w-hat1,
w-hat2, all the way to w-hatT.
And once we learn them, making a prediction is very similar
to what you do for logistic regression or a linear classifier.
It's just the sign of the weighted sum of the votes from each classifier.
And so if you look at this, it looks a lot like a linear classification model,
logistic regression and all of those, exactly the same.
However, not only are we learning the ws from data,
we're actually learning the features.
In those models, we had hs to represent our features.
Here, the feature are these fs,
which are the weight classifiers that we're going to learn from data.
So we can think about boosting as an approach to learn features from data.
And it's really super exciting.
[MUSIC]