[MUSIC] I'd like to take a moment now to
summarize we've seen boosting and what impacts it's had in the real world. Let's talk about AdaBoost. So AdaBoost is one of the early
types of boosting algorithms. Extremely useful, but
there other algorithms out there. In particular,
there's one called gradient boosting, which is slightly more complicated but
extremely similar. And it's like AdaBoost, but
it can be useful not not just for basic classification, but for
other of types of loss functions for the types of problems and it's what
most people use, so gradient boosting. It's kind of generalization kind of boost,
you can think about it that way. Then there's other related
ways to learn ensembles, the most popular one is
called random forests. So random forest is a lot like boost
in the sense that it tries to learn an example of classifiers in this case,
decision trees, but it could be other types of classifiers. And instead of using boosting,
it uses an approach called bagging. And just very briefly, what you do with
bagging is, you take your data set, and you sample different subset of the data,
which is kind of like learning on different sub datasets, and we learn
the decision tree on each one of them. You just average the outputs. So you're not optimizing
the coefficients that we had, and we're learning from
different subset of data. It's easier to parallelize, but it tends
not perform as well as boosting for a fixed number of trees. So for 100 trees, or 100 serial stumps, boosting tends to perform
better than random forest. Now let's take a moment to discuss impact boosting has had in
the machine learning world. And hint, hint, it's been huge. It's amongst the most useful machine
learning approaches out there. It's useful in a wide range of fields so,
for example, in computer vision a lot of the default algorithm in
computer vision is boosting, like face detection algorithms
where you take your camera, you point it at something and
tries to detect your face. Of which their that uses
boosting very useful. If you look at machinery competition
they've become very popular the last two or three years from places
like Kaggle or KDD Cup. Most winners, so this is more than half, I think it's about 70% of winners actually
use boosting to win their competition. If fact, they use Boosta Freeze, and
this looks at wide range of tasks like malware detection or
fraud detection or ranking web searches, and even interesting physics tasks
like detecting the Higgs Boson. All those problems and all those challenges have been
won by boosting decision trees. And this is perhaps one of
the most deployed advanced machinery methods out there. Particularly the notion of ensembles. So for example,
if you know about Netflix which is an online place we can
watch movies online. This kind of company, they recommend
what movie you might want to watch next. That system uses boosting. Actually uses an example of crossfires. But more interestingly, they had a
competition a few years ago, where people tried to provide better recommendations
and the winner was one that treated assemble of many, many, many classifiers
in order to create better recommendations. So assembles, you'll see them everywhere. Sometimes they optimize boosting, sometimes they optimize with
different techniques like bagging. And sometimes people just by
hand tuned away to say okay, I'm going to give you one to this,
half to that. I don't recommend the last approach,
I recommend boosting as a one to use. Great, so in this module we've explored
the notion of an ensemble classifier and we formalized ensembles as the way to
combination of the loads of different classifiers and we discussed generally
the boosting algorithm where the next classifier focuses on
the mistakes that we made so far. As well as Adaboost,
which is a special case for classification where we show you how to come up with
the coefficients of each classifier and the weights on the data. We've discussed how to
implement decision stumps, with the decision stumps,
extremely easy to do. And then talked a little bit about
the conversions property of how the AdaBoosting tends to go to 0, but you
have to be concerned a little bit about the over 50, although it tends to
be a robust over 50 in practice. [MUSIC]