Optimizers in Deep Learning: SGD, RMSProp, and Adam

[00:00:00]Welcome back. Training neural networks with adaptive learning rate.

[00:00:07]Okay. Now, neural networks are trained using erm gradient descent.

[00:00:30]Now if uh suppose um rcap of theta is the empirical risk.

[00:00:44]Now gradient descent estimates the parameters as follows. Now you have uh theta t + 1 to be theta t minus alpha * the gradient of the empirical risk with respect to theta.

[00:01:05]Right? This is what the first order gradient descent is all about. Uh there are versions to it. Right? One version is what is called as the stoastic gradient descent.

[00:01:27]where compute uh r tilda of theta as estimate the loss on on a set of samples, right? Which is uh where where B is a subset of the data set D randomly sampled.

[00:02:06]We already saw that this has a regularization effect is what I told you. So this is called stoastic gradient descent or SGD where if you have thousand samples you construct a batch of 32 samples or 64 samples which is again a hyperparameter and then perform the gradient descent and one full um walk through the entire training data is called an epoch during training. Okay, an iteration is where you do one forward pass and one backward pass through uh the neural network that you are training. If you do a stoastic gradient descent and uh pass through the entire data set once then it's called an epoch and training is done for several epochs.

[00:02:48]That's how the training is done. Okay, this is stoastic gradient descent. Uh but in in both in stoastic gradient descent and the gradient descent, right, the step size, okay, alpha is fixed.

[00:03:01]This is also called learning rate.

[00:03:08]This learning rate is fixed in stoastic gradient descent. You fix it to a particular value. Okay. And you keep it the same. But uh in optimization we know that while we are looking at uh gradient descent or rather numerical algorithms we may have to take a larger step at times where the gradient has uh uh larger magnitude and you are near the uh the optima we have to take a smaller step uh at other times and so on. So basically instead of having the fixed learning rate, it's always a better idea to have adaptive learning rate. The learning rate that adapts. Okay. So there are techniques that would incorporate this. There are a few adaptive learning rate techniques for gradient descent.

[00:04:09]we look at uh uh a few famous uh such things. Uh the first thing is called um SGD with momentum where the idea is as follows. Define momentum mt as beta 1 * mt - 1 + 1 - beta 1 * gt where so gt is the gradient of the loss function.

[00:05:05]And now you make theta t + 1 as theta t minus some alpha * mt.

[00:05:24]Yeah. What is this doing?

[00:05:28]This MT is a running average of the gradients, moving average of the gradients at t= 1. It is simply the gradient.

[00:05:40]Okay. At when you initialize mt m0 to zero, so it's equal to the gradient. As you keep moving in time, the m the the m term will keep accumulating the gradients and taking an average of it as we move on.

[00:05:58]Okay. Now, instead of subtracting the the the gradients, you subtract the accumulated gradients over time. Why is this a good idea?

[00:06:11]You're computing the exponential decaying average of gradients?

[00:06:16]Which means in earlier times you take larger steps depending upon the gradients and the later steps you take smaller steps. the the gradient because the gradients are being accumulated over time.

[00:06:32]So this is called SGD with momentum and this term MT is called the momentum term.

[00:06:38]Okay. The see here what did we do is that we accumulated the gradients but we know that in optimization uh the first order gradients are not enough right? you need to accumulate the second order terms of the gradients as well.

[00:06:57]Why? Gives you the acceleration effect.

[00:07:02]Right? So instead of only looking at the first order gradients or rather uh the gradients, look at the gradient squared terms as well. Okay? So this is called there's another adaptive learning rate method which is called RMS prop.

[00:07:22]So here instead of averaging the gradient you average the second order or rather the squared of the gradient define wt to be some beta 2 * vt - 1 + 1 - beta 2 * gt² we are accumulating the square root of gradients here now you make theta t + 1 to be theta t minus some alpha divided by roo<unk> of v_t into gt.

[00:08:05]So why do we need this root of wt? V_t the dimensions of the v_t and not the dimensions r dimensions it's the physical dimensions of gt is square root of gradients right. So root of wt has the dimensions as that of gradients and gt is gradients they both cancel and the one that this will be of dimensions of the parameters and that's why you can add and subtract them the dimensionality has to match right yeah so that's why this is the uh the rms prop where what we are doing is that instead of using the first order gradients or rather the the gradients we're using the gradient squared terms Okay, why do we have to do this?

[00:08:50]See, we are scaling the gradients. All gradients are now being scaled with a particular value. This particular value is inverse of their squared.

[00:09:01]So the gradients, okay, are the parameters for which the gradients are damping uh higher the gradients, they dampen quicker.

[00:09:12]The lower the gradient values they they dampen slower. So here what we did was we dampened all the gradients no matter what their the so-called acceleration is. Here they are being scaled by the inverse of the uh damping factor or the acceleration. That's it.

[00:09:31]Okay. This is about RMS prop. Uh the other update to this improvisation of this is what is called as adaptive momentum estimation also known as Adam.

[00:10:01]And by the way, even though I've written it this way, right? Uh if theta are vectors, these are done at the element level, the parameter level. I mean these things are actually the hadamar products because when you take the square root of gradient, you you do it dimension wise.

[00:10:20]Each of the dimension has to be multiplied. So you please note that if you do this then there there is a different learning rate for different parameter also.

[00:10:34]You understand there is adaptability across different parameters. You know some parameters are are updated faster or larger and some parameters are adapt are are are changed slower because this is happening at the parameter level because these are vectors. No, we are looking at haramat products here. Do you see that?

[00:10:56]Okay. So, this adaptive momentum estimation which is referred to as Adam actually combines both. It's a combination of both the RMS prop and the HTD momentum. There we have both the terms. So compute momentum as beta 1 * mtus 1 + 1 - beta 1 * gt and uh beta 2 * vt - 1 + 1 - beta 2 * gt² and the final estimate Right.

[00:11:38]Theta t + 1 is theta t minus alpha divided by roo<unk> of v_t times the momentum.

[00:11:49]So this is a combination of both the rms prop and uh the momentum term. You have both the momentum and the second order term. This is called atom. Now if you go to the standard optimizers such as pyarch etc. you have the uh the facility to choose your optimizer. You can choose SGD, you can choose RMS prop, you can choose SGD with momentum or atom. Adam is the go-to choice for optimizers in today's uh learning because empirically it has been observed to be uh leading to good convergence. Now you have to choose this beta 1 and beta 2. They are hyperparameters.

[00:12:30]Okay, adaptive momentum estimation. Uh, now that we are talking about uh optimizing neural networks, there is one point that I would uh I would make uh which is which is uh better to be known.

[00:12:43]See, we are using gradient descent here.

[00:12:47]Okay. To optimize over the parameters parameter space that would be typically a very large size, right? I mean today they are of billions of parameters you know there can be thousands of so we are optimizing in a very large dimensional space and the objective function that we are that is being optimized is not a convex function of these parameters.

[00:13:12]The empirical risk that we are optimizing is no uh convex function.

[00:13:17]Okay. So first order gradient descent does not guarantee that that you will go to the true optima in in fact it can be is that the the the likelihood of these methods taking you to uh the true minimizer is very very low and it can be shown mathematically. Let me just tell you that.

[00:13:42]So let's say that we are operating in a thousand dimensional space.

[00:13:47]Our neural network has thousand per thousand is nothing. You know a simple uh ant level MLP will also have thousand par more than thousand parameters. Let's say that you have a neural network that has thousand parameters.

[00:14:00]Now suppose uh there is an optimization surface. When do you know or how do you find out whether a given point on this optimization surface is a point of extrema or not?

[00:14:14]Recall your uh high school or uh college math. How gradient has to be zero.

[00:14:20]Right? That will only tell you that the gradient vanishes at that point. How do you know that if it's a maximum or a minima?

[00:14:28]Second derivative test. You'll have to always look at the second derivative test. Now if you are working with uh vector valued functions what is the equivalent of the second derivative test?

[00:14:39]>> Not the hessian. Hessian is the equivalent to second derivative. But if if a point has to be a global optima then the signs of the igon values of the hessian matrix are all have to be of the same sign. If all of them are positive or in in in other words if the hessian matrix is positive definite then that particular point is a minima or maxima >> yeah whatever right it has to be it has to be all the values have to be of the exact same sign now let's say that we are operating in a thousand dimensional space what is the size of the hashian >> which is 10 to six a million we have We are looking at a million uh >> igon values. Looking at million values.

[00:15:27]Let's say that we model the sign of the igon value as a bernoli random variable with success probability p.

[00:15:37]What am I saying? With probability p the sign of the igon value of the hashian matrix is positive. with probability 1 minus P. The success probability is that the probability that the sign of the igon value of the H matrix is negative is 1 minus P. So now take this P to be.99999.

[00:15:58]So with a very high probability the value of the session matrix is going to be positive.

[00:16:05]Now what is.99 to the power of 10 to the 6?

[00:16:12]No.99 to the^ of 10 to 6 is a very very small number right which means that the odds of any point that you land on to on the loss surface in a thousand dimensional space being a global optima is extremely low.

[00:16:32]This by this with an assumption that the probability of uh it taking a positive value is.99.

[00:16:41]So what am I trying to say? So now we are looking at not at thousand parameter neural networks. We are looking at millions and billions of parameters of neural network. So almost always gradient descent will not take you to the point that is a global optima. Then what is the hope? The hope is look at your validation data. The only question that all the points that we land uh to with gradient descent are local optimas.

[00:17:10]The only question that is to be asked is is your local optima better than my local optima.

[00:17:16]And how do we measure this? By looking at validation data. That's why when you try neural networks, there is no theoretical way to look at what the stopping criteria is.

[00:17:28]Okay. Just you have that is why people look at come up with benchmarks. You have test data. Whatever neural network you train in performance or rather uh extract its performance on test data and that's that's your uh truth that's it okay in fact there is one other nomure that is used when you are doing gradient descent right every single point of gradient descent will give you a different model because you are navigating in the theta space correct and these things are called checkpoints by the way so what is shipped are checkpoints model checkpoints A model checkpoint is nothing but a particular set of theta for this architecture. So when you download a model a pre-trained model from some of these uh commercially available uh libraries hugging face for instance what they actually give you are these thetas with an erm done on a particular task start from it either do inference on it or you can initialize another neural network with that and you know you do another erm on top of it or do distillation or do whatever you want is okay so I think now we are in a position to take up a data set take any neural network and train it to for whatever task you want. This is the foundational principle of training any uh you know neural network for any task including LLMs and today's LLMs are trained on thousands of uh or not thousands hundreds of thousands of GPU cards for several tens of months on trillions of tokens of data that's the scale >> with billions of parameters exactly right that is what it is Right. Okay. So this completes our discussion on neural networks. We will next go to uh the class classification and regression trees, decision trees and subsequently we look at ensemble methods. Thank you.