Optimizers in Deep Learning: SGD, RMSProp, and Adam

Added:

The Generalization Gap of Adaptive Optimizers: The Resurgence of SGD

While adaptive optimizers like Adam and RMSProp offer rapid convergence during training, they often generalize poorly to unseen test data compared to standard Stochastic Gradient Descent (SGD) with momentum. This phenomenon is known as the 'generalization gap.' Adaptive methods scale learning rates based on historical gradients, which can cause the optimizer to overfit to specific features and settle into 'sharp' local minima that perform poorly on test sets. Consequently, many state-of-the-art models in computer vision and natural language processing still rely on finely tuned SGD for their final training runs. This critical limitation has sparked a research shift toward hybrid optimizers (like AdamW) and second-order optimization methods (like K-FAC) that aim to combine the fast initial convergence of adaptive methods with the superior generalization capabilities of SGD.

Fundamental concept of Gradient Descent, including how a loss function is minimized.
Basic understanding of Neural Network architecture and the Backpropagation algorithm.
Multivariable Calculus concepts, specifically partial derivatives and the gradient vector.
The role of the Learning Rate hyperparameter and the challenges of manual tuning, such as overshooting or slow convergence.
Advanced optimizer variants such as AdamW (decoupled weight decay) and AdaBelief.
Learning Rate Scheduling techniques, including Cosine Annealing and One-Cycle policies, to dynamically adjust rates during training.
Hyperparameter optimization strategies for tuning optimizer-specific parameters like beta coefficients and epsilon.
The practical implications of batch size on optimization stability, convergence speed, and generalization capability.
106 views5likes19:28@nptel-indianinstituteofsci8064Original Release: 2026-03-20

This lecture explains adaptive learning rate optimization methods for training neural networks, including SGD with momentum (which uses an exponential moving average of gradients), RMSProp (which scales gradients by the inverse square root of accumulated squared gradients), and Adam (which combines both momentum and RMSProp principles). The lecture emphasizes that in high-dimensional non-convex optimization problems common in neural networks, gradient descent rarely reaches global optima, making adaptive optimizers essential for practical training. Adam has become the go-to optimizer in modern deep learning due to its empirical effectiveness across various architectures.