Ridge Regression Explained: L2 Regularization for Overfitting

[00:00:00]regularization it's just another way to save desensitization let's check it out with a new regression stat quest hello I'm Josh stormer and welcome to stat quest today we're going to do part 1 of a series of video on regularization techniques in this video we're gonna cover Ridge regression and it's going to be clearly explained note this stat cuesta seems you understand the concepts of bias and variance in the context of machine learning if not check out machine learning fundamentals bias and variance it also assumes that you are familiar with linear models if not check out the following stat quests the links are in the description below lastly if you're not already familiar with the concept of cross-validation check out the stack west on cross-validation in this stack quest we will one look at a simple example that shows the main ideas behind Ridge regression to go into details about how Ridge regression works three show how Ridge regression works in a variety of situations and four lastly we'll talk about how Ridge regression can solve the unsolvable BAM let's start by collecting weight and size measurements from a bunch of mice since these data look relatively linear we will use linear regression aka least squares to model the relationship between weight and size so we'll fit a line to the data using least squares in other words we find the line that results in the minimum sum of squared residuals ultimately we end up with this equation for the line the line has two parameters a y-axis intercept and a slope we can plug in a value for weight for example two point five and do the math and get a value for size together the value for weight two point five and the value for size two point eight give us a point on the line when we have a lot of measurements we can be fairly confident that the least squares line accurately reflects the relationship between size and weight but what if we only have two measurements we fit a new line with least squares since the new line overlaps the two data points the minimum sum of squared residuals equals zero ultimately we end up with this equation for the new line note here are the original data in the original line for comparison let's call the two red dots the training data in the remaining green dots that testing data the sum of the squared residuals for just the two red points the training data is small in this case it is zero but the sum of the squared residuals for the green points the testing data is large and that means that the new line has high variance in machine learning lingo we'd say that the new line is over fit to the training data now let's go back to just the training data we just saw that least squares results in a line that is over fit and has high variance the main idea behind Ridge regression is to find a new line that doesn't fit the training data as well in other words we introduce a small amount of bias into how the new line is fit to the data but in return for that small amount of bias we get a significant drop in variance in other words by starting with a slightly worse fit Ridge regression can provide better long-term predictions BAM now let's dive into the nitty-gritty and learn how Ridge regression works let's go back to just the training data when least-squares determines values for the parameters in this equation it minimizes the sum of the squared residuals in contrast when Ridge regression determines the values for the parameters in this equation it minimizes the sum of the squared residuals plus lambda times the slope squared note I usually try to avoid using Greek characters as much as possible but if you are ever going to do Ridge regression in practice you have to know that this term is called lambda this part of the equation adds a penalty to the traditional least squares method and lambda determines how severe that penalty is to get a better idea of what's going on let's plug in some numbers let's start by plugging in the numbers that correspond to the least squares fit the sum of the squared residuals for the least squares fit is zero because the line overlaps the data points and the slope is one point three we'll talk more about lambda later but for now let lambda equal one all together we have zero plus one times one point three squared and when we do the math we get one point six nine now let's see what happens when we plug in numbers for the ridge regression line the sum of the squared residuals is zero point three squared for this residual plus zero point one squared for this residual the slope is 0.8 and just like before we'll let lambda equal 1 altogether we have 0.3 squared plus 0.1 squared plus 1 times 0.8 squared and when we do the math we get 0.7 for for the least squares line the sum of squared residuals plus the ridge regression penalty is one point six nine for the ridge regression line the sum of squared residuals plus the ridge regression penalty is 0.74 thus if we wanted to minimize the sum of the squared residuals plus the ridge regression penalty we would choose the ridge regression line over the least squares line without the small amount of bias that the penalty creates the least squares fit has a large amount of variance in contrast the ridge regression line which has a small amount of bias due to the penalty has less variance now before we talk about lambda let's talk a little bit more about the effect that the ridge regression penalty has on how the line is fit to the data to keep things simple imagine we only have one line this line suggests that for every one unit increase in weight there is a one unit increase in predicted size if the slope of the line is steeper than for every one unit increase in weight the prediction for size increases by over two units in other words when the slope of the line is steep then the prediction for size is very sensitive to relatively small changes in weight when the slope is small then for every one unit increase in weight the prediction for size barely increases in other words when the slope of the line is small then predictions for size are much less sensitive to changes in weight now let's go back to the least squares and Ridge regression lines fit to the two data points the ridge regression penalty resulted in a line that has a smaller slope which means that predictions made with the ridge regression line are less sensitive to weight than the least squares line BAM now let's go back to the equation that Ridge regression tries to minimize and talk about lambda lambda can be any value from 0 to positive infinity when lambda equals zero then the ridge regression penalty is also zero and that means that the ridge regression line will only minimize the sum of squared residuals and the ridge regression line will be the same as the least squares line because they are both minimizing the exact same thing now let's see what happens as we increase the value for lambda in the example we just looked at we said lambda equals 1 and the ridge regression line ended up with a smaller slope than the least squares line when we set lambda equals 2 the slope gets even smaller and when we set lambda equals 3 the slope is even smaller and the larger we make lambda the slope gets asymptotically close to 0 so the larger lambda gets our prediction for size become less and less sensitive to weight so how do we decide what value to give lambda we just try a bunch of values for lambda and use cross-validation typically 10-fold cross-validation to determine which one results in the lowest variance double bail in the previous example we showed how ridge regression would work when we want to predict size which is a continuous variable using weight which is also a continuous variable however Ridge regression also works when we use a discrete variable like normal diet versus high fat diet to predict size in this case the data might look like this in the least squares fitted equation might look like this where 1.5 the equivalent of a y-intercept corresponds to the average size of the mice on the normal diet and 0.7 the equivalent of a slope corresponds to the difference between the average size for the mice on the normal diet compared to the mice on the high-fat diet note from here on out we'll refer to this distance as diet difference high-fat diet is either zero for mice on a normal diet or one for mice on the high-fat diet in other words this term alone predicts the size of mice on the normal diet in the sum of these two terms is the prediction for the size of mice on the high-fat diet for the mice on the normal diet the residuals are the distances between the mice and the normal diet mean and for mice on the high-fat diet the residuals are the distances between the mice and the high fat diet mean when Lee squares determines the values for the parameters in this equation it minimizes the sum of the squared residuals in other words these distances between the data and the means are minimized when Ridge regression determines values for the parameters in this equation it minimizes the sum of the squared residuals plus lambda times diet difference squared remember diet difference simply refers to the distance between the mice on the normal diet and the mice on the high-fat diet when lambda equals zero this whole term ends up being zero and we get the same equation that we got with least squares but when lambda gets large the only way to minimize the whole equation is to shrink diet distance down in other words as lambda gets larger our prediction for the size of mice on the high-fat diet becomes less sensitive to the difference between the normal diet and the high-fat diet and remember the whole point of doing rich regression is because small sample sizes like these can lead to poor least squares estimates that result in terrible machine learning predictions BAM Ridge regression can also be applied to logistic regression in this example we are using weight to predict if a mouse's obese or not this is the equation for this logistic regression and Ridge regression would shrink the estimate for the slope making our prediction about whether and out of mouse is obese less sensitive to weight note when applied to logistic regression Ridge regression optimizes the sum of the likelihoods instead of the squared residuals because logistic regression is solved using maximum likelihood so far we've seen simple examples of how Ridge regression helps reduce variance by shrinking parameters and making our predictions less sensitive to them but we can apply Ridge regression to complicated models as well in this model we've combined the weight measurement data from the first example with the two diets from the second example combining these two datasets gives us this equation and Ridge regression tries to minimize this now the ridge regression penalty contains the parameters for the slope and the difference between diets in general the ridge regression penalty contains all of the parameters except for the y intercept if we had a big huge crazy equation with terms for astrological sign the airspeed of a swallow and other stuff then the ridge regression penalty would have all those parameters squared except for the y intercept every parameter except for the y intercept is scaled by the measurements and that's why the y intercept is not included in the ridge regression penalty double bam okay now the next thing we're going to talk about is going to sound totally random but trust me it will lead to the coolest thing about Ridge regression it's so cool it's almost like magic we all know that this is the equation for a line in an order for least squares to solve for the parameters the y-intercept and slope we need at least two data points these data points result in these parameters in this specific line if we only have one data point then we wouldn't be able to solve for these parameters because there would be no way to tell if this line is better than this line or this line or any old line that goes through the one data point all of these lines have zero residuals and thus all minimize the sum of the squared residuals it's not until we have two data points that it becomes clear that this is the least square solution now let's look at an equation that has three parameters to estimate we need to estimate a y-intercept a slope that reflects how weight contributes to the prediction of size and a slope that reflects how age contributes to the prediction of size when we have three parameters to estimate then just two data points isn't going to cut it that's because in three dimensions which is what we get when we add another access to our graph for age we have to fit a plane to the data instead of just a line and with only two data points there's no reason why this plane fits the data any better than this plane or this plane but as soon as we have three data points we can solve for these parameters if we have an equation with four parameters then least squares needs at least four data points to estimate all four parameters and if we have an equation with 10,001 parameters then we need at least 10,000 and one data points to estimate all of the parameters an equation with 10,001 parameters might sound bonkers but it's more common than you might expect for example we might use gene expression measurements from 10,000 genes to predict size and that would mean we would need gene expression measurements from ten thousand and one mice unfortunately collecting gene expression measurements from ten thousand and one mice is crazy expensive and time-consuming right now in practice a huge data set might have measurements from 500 mice so what do we do if we have an equation with ten thousand and one parameters in only 500 data points we use Ridge regression it turns out that by adding the ridge regression penalty we can solve for all 10,000 won parameters with only 500 or even fewer samples one way to think about how Ridge regression can solve for parameters when there isn't enough data is to go back to our original size versus weight example only this time there is only one data point in the training set least squares can't find a single optimal solution since any line that goes through the dot will minimize the sum of the squared residuals but Ridge regression can find a solution with cross-validation and the Ridge regression penalty that favors smaller parameter values since this stack quest is already super long we'll save a more thorough discussion of how this works for a future stat quest triple bam in summary when the sample sizes are relatively small then Ridge regression can improve predictions made from new data ie reduce variance by making the predictions less sensitive to the training data this is done by adding the ridge regression penalty to the thing that must be minimized the ridge regression penalty itself is lambda times the sum of all squared parameters except for the y-intercept and lambda is determined using cross-validation lastly even when there isn't enough data to find the least squares parameter estimates Ridge regression can still find a solution using cross-validation and the ridge regression penalty hooray we've made it to the end of another exciting stat quest if you liked this stack quest and want to see more please subscribe and if you want to support stack quest well consider buying one or two of my original songs alright until next time quest on