Return to Video

Regularization - Intro to Machine Learning

  • 0:01 - 0:05
    One very powerful place that you can use regularization, is in regression.
  • 0:05 - 0:07
    Regularization is a method for
  • 0:07 - 0:11
    automatically penalizing the extra features that you use in your model.
  • 0:11 - 0:13
    So, let me make this a little bit more concrete.
  • 0:13 - 0:17
    There's a type of regularized regression called Lasso Regression.
  • 0:17 - 0:20
    And, here's the rough formula for the Lasso Regression.
  • 0:21 - 0:22
    A regular linear regression would say,
  • 0:22 - 0:28
    I just want to minimize the sum of the squared errors in my fit.
  • 0:28 - 0:31
    I want to minimize the distance between my fit, and
  • 0:31 - 0:35
    any given data point, or the square of that distance.
  • 0:35 - 0:40
    What Lasso Regression says is yeah, we want a small sum of squared error.
  • 0:40 - 0:43
    But, in addition to minimizing the sum of the squared errors,
  • 0:43 - 0:46
    I also want to minimize the number of features that I'm using.
  • 0:46 - 0:49
    And, so I'm going to add in a second term here, in which I
  • 0:49 - 0:54
    have a penalty parameter, and I have the coefficients of my regression.
  • 0:54 - 0:58
    So, this is basically the term that describes how many features I'm using.
  • 0:58 - 1:00
    So, here's the result of this formulation.
  • 1:00 - 1:05
    When I'm performing my fit, I'm considering both the errors that come from that
  • 1:05 - 1:09
    fit, and also the number of features that are being used.
  • 1:09 - 1:11
    And, so let's say I'm comparing two different fits,
  • 1:11 - 1:13
    that have different number of features in them.
  • 1:14 - 1:16
    The one that has more features included,
  • 1:16 - 1:19
    will almost certainly have a smaller sum of the squared error.
  • 1:19 - 1:23
    because, it can fit more precisely to the points.
  • 1:23 - 1:25
    But, I pay a penalty for using that extra feature.
  • 1:25 - 1:29
    And, that comes in the second term with the, with the penalty term, and
  • 1:29 - 1:31
    the coefficients of regression that I'm going to get for
  • 1:31 - 1:34
    that additional feature that I'm using.
  • 1:34 - 1:37
    And, so what this is saying is that the gain that I get,
  • 1:37 - 1:39
    in terms of the, the precision,
  • 1:39 - 1:44
    the goodness of fit of my regression, has to be a bigger gain than the, the loss
  • 1:44 - 1:48
    that I take as a result of having that additional feature in my regression.
  • 1:49 - 1:53
    So, this precisely formulates, in a mathematical way, the trade off between
  • 1:53 - 1:59
    having small errors and having a simpler fit that's using fewer features.
  • 1:59 - 2:01
    And, so what Lasso Regression does,
  • 2:01 - 2:05
    is it automatically takes into account this penalty parameter.
  • 2:05 - 2:09
    And, in so doing, it helps you actually figure out which features that
  • 2:09 - 2:12
    are the ones that have the most important effect on your regression.
  • 2:12 - 2:15
    And, once it's found those features, it can actually eliminate or
  • 2:15 - 2:20
    set to zero, the coefficients for the features that basically don't help
Title:
Regularization - Intro to Machine Learning
Description:

more » « less
Video Language:
English
Team:
Udacity
Project:
ud120 - Intro to Machine Learning
Duration:
02:21
Udacity Robot edited English subtitles for 11-19 Regularization
Udacity Robot edited English subtitles for 11-19 Regularization
Cogi-Admin edited English subtitles for 11-19 Regularization

English subtitles

Revisions Compare revisions