Academic Databases
Explore Our Semantic Databases
STEM & Tech
Math, Physics, CS & Engineering...
Economics
Macro, Micro, Econometrics...
Business
Management, Marketing, Finance...
Health & Life Sciences
Biology, Medicine, Neuroscience...
Humanities & Social Sciences
Psychology, Law, Philosophy, History...
General Studies
General & interdisciplinary studies...
Explore Topics
Features

Backpropagation Explained: Neural Network Optimization Basics

Added: 2026-10-07

Deep Dive

Basic architecture of artificial neural networks, including inputs, weights, biases, and activation functions.

What is ANN learning, Hebb & Perception Learning (Lecture-8 on Neural Network and Fuzzy Logic)

What is ANN learning, Hebb & Perception Learning (Lecture-8 on Neural Network and Fuzzy Logic)

2.1K

An artificial neural network consists of an input layer, output layer, and optionally hidden layers. Inputs (x1, x2, ..., xn) are accumulated in an accumulator along with a bias term. Each input has an associated weight (w1, w2, ..., wn) representing the strength of that input. The accumulated net value passes through an activation function (such as sigmoid, identity, or step function) to produce the desired output.

Click to watch →

Reliability of Artificial Intelligence - Chances and Challenges: Gitta Kutyniok (LMU Munich)

Reliability of Artificial Intelligence - Chances and Challenges: Gitta Kutyniok (LMU Munich)

253

Artificial neural networks consist of artificial neurons with inputs X₁ to Xₙ, weights, bias, and activation functions. Weights and biases are learned during the training process. The network produces a highly structured mathematical function composed of linear terms and activation functions, typically ReLU. This architecture was originally inspired by McCulloch and Pitts' 1943 work to mimic the human brain.

Click to watch →

Lec2: Introduction: Artificial Neural Networks(Perceptron)

Lec2: Introduction: Artificial Neural Networks(Perceptron)

2.2K

Neural network architecture consists of input nodes representing features, weights (synaptic weights) that determine the importance of each input, and a bias term. The weights are adjustable parameters that change during learning to optimize the network's performance. The bias term allows the activation function to be shifted, providing additional flexibility in the model.

Click to watch →

[Kecerdasan Buatan] Artificial Neural Network : Backpropagation | yutikamelia

[Kecerdasan Buatan] Artificial Neural Network : Backpropagation | yutikamelia

224

Artificial Neural Networks (ANN) consist of three main components: (1) Input layer that receives data signals (X1, X2, X3, ..., XP), (2) Weight values (W) associated with each input that determine the importance of each input, and (3) A summation function (sum function) that combines the weighted inputs. The network may also include a bias term (B) that is added to the weighted sum. The output is generated after processing through the activation function.

Click to watch →

Lect1 ANN

Lect1 ANN

562

An artificial neuron unit consists of input linkage, input function, activation function, and output linkage. Weight factors represent connection strength between units, initially set to zero and increasing to 1 through repeated learning. Learning uses training examples where specific inputs must produce expected outputs. Neural networks consist of input units, hidden units, and output units connected by weighted links. These weight factors can be systematically represented in matrix form. Neural networks are classified as linear (calculating weighted sums without activation functions) or non-linear (including activation functions). Three common activation functions are: step function (0 below threshold, 1 above), sine function (-1 for negative, +1 for positive inputs), and sigmoid function (1/(1 + e^(-x)) producing values between 0 and 1). The choice of activation function depends on input nature and calculation requirements.

Click to watch →

Deep Dive

Multivariate calculus, specifically the chain rule for computing partial derivatives.

Chain Rule For Multivariable Functions | Calculus 3 Lesson 47 - JK Math

Chain Rule For Multivariable Functions | Calculus 3 Lesson 47 - JK Math

3.5K

Apply the chain rule for partial derivatives by computing all required partial derivatives of the multivariable function with respect to intermediate variables, then computing partial derivatives of intermediate variables with respect to each independent variable. Multiply corresponding pairs and sum them. For example, with w = x³ + y³, x = t² + s², y = 3ts: ∂w/∂x = 3x², ∂w/∂y = 3y², ∂x/∂s = 2s, ∂x/∂t = 2t, ∂y/∂s = 3t, ∂y/∂t = 3s. Then ∂w/∂s = (3x²)(2s) + (3y²)(3t) and ∂w/∂t = (3x²)(2t) + (3y²)(3s).

Click to watch →

CHAIN RULE IN PARTIAL DIFFERENTIATION AND MUILTIVARIABLE CALCULUS SOLVED PROBLEM 6 @TIKLESACADEMY

CHAIN RULE IN PARTIAL DIFFERENTIATION AND MUILTIVARIABLE CALCULUS SOLVED PROBLEM 6 @TIKLESACADEMY

1.2K

The chain rule is fundamental in multivariable calculus for computing derivatives of composite functions. The process involves: (1) identifying the function and its variables, (2) constructing a chain showing relationships between variables, (3) applying the chain rule multiple times for multiple variables, and (4) computing partial derivatives. For example, if u = f(R, S) where R and S are functions of x, y, z, we construct the chain u → RS → x, y, z. Each application follows the pattern: direct path plus step-by-step paths. Before differentiating complex expressions, simplify them first (e.g., R = (y-x)/xy becomes R = 1/x - 1/y). This systematic approach enables accurate computation of partial derivatives.

Click to watch →

Grings - Regra da cadeia função de duas variáveis aula 3

Grings - Regra da cadeia função de duas variáveis aula 3

127K

When applying the chain rule to multivariable functions, you compute partial derivatives treating other variables as constants. For example, if z = x² + 2xy, then ∂z/∂x = 2x + 2y (treating y as constant) and ∂z/∂y = 2x (treating x as constant). These partial derivatives are then multiplied by the respective derivatives of x and y with respect to the independent variable.

Click to watch →

Multivariable Chain Rule

Multivariable Chain Rule

1.1K

To apply the multivariable chain rule, first compute the partial derivatives of the outer function z with respect to x and y. For z = e^(xy)cos(x²), the partial with respect to x uses the product rule: ∂z/∂x = e^(xy)cos(x²) + e^(xy)(-sin(x²)(2x)). The partial with respect to y is simpler: ∂z/∂y = xe^(xy)cos(x²).

Click to watch →

Differentiation | Chain Rule | Multivariate Function

Differentiation | Chain Rule | Multivariate Function

855

To apply the chain rule, compute partial derivatives while holding other variables constant. ∂U/∂X = 2X (treating Y as constant). ∂U/∂Y = 2 - 2Y (treating X as constant). ∂X/∂R = cos(θ) (treating θ as constant). ∂Y/∂R = sin(θ) (treating θ as constant). These partial derivatives are then multiplied and summed according to the chain rule formula.

Click to watch →

Deep Dive

The concept of a loss (or cost) function, such as Mean Squared Error, and its role in evaluating model error.

1-Mavzu: Optimallashtirish nima va nega kerak?

1-Mavzu: Optimallashtirish nima va nega kerak?

162

This section explains the loss function as the fundamental tool for measuring model error in machine learning. The instructor defines the loss function as a formula that quantifies how much a model's predictions differ from actual values, similar to how a thermometer measures temperature. The Mean Squared Error (MSE) is presented as the primary example, where the loss equals the squared difference between predicted and actual values. The instructor demonstrates through concrete examples: a $5,000 error produces a loss of 25 million, while a $10,000 error produces a loss of 100 million. The squaring operation is emphasized as a mechanism that heavily penalizes larger errors, making the loss function sensitive to significant deviations.

Click to watch →

Cost Function Machine Learning | Cost Function In Machine Learning With Example | Intellipaat

Cost Function Machine Learning | Cost Function In Machine Learning With Example | Intellipaat

4.4K

A cost function (or loss function) is a mathematical function that quantifies the error between predicted values and actual values in a machine learning model, enabling model evaluation and optimization; the three main types are Mean Absolute Error (MAE), which calculates the average of absolute differences; Mean Squared Error (MSE), which calculates the average of squared differences and penalizes larger errors more heavily; and Root Mean Squared Error (RMSE), which is the square root of MSE and provides a balanced penalty for large errors while keeping the output magnitude manageable.

Click to watch →

133 - What are Loss functions in machine learning?

133 - What are Loss functions in machine learning?

97.7K

A loss function, also called a cost function or error function, quantifies the error between the model's output and the target value in training data. For example, if the true value is 10 and the model predicts 9.7, the loss function measures this difference to evaluate how well the model is performing.

Click to watch →

Función de coste: error cuadrático medio | Machine Learning 101

Función de coste: error cuadrático medio | Machine Learning 101

2.6K

The Mean Squared Error (MSE) is a cost function used in machine learning to measure the average squared difference between predicted and actual values in regression models. It works by calculating the distance between each predicted value and its corresponding actual value, squaring this difference to eliminate negative signs and emphasize larger errors, then averaging across all data points. This function serves dual purposes: it evaluates model performance by quantifying prediction error, and it guides the training process by helping algorithms iteratively adjust model parameters to minimize the error, ultimately finding the optimal parameters that best fit the training data.

Click to watch →

🔍 ما هي الـ Cost Function؟ فهم دالة الخطأ في تعلم الآلة خطوة بخطوة!

🔍 ما هي الـ Cost Function؟ فهم دالة الخطأ في تعلم الآلة خطوة بخطوة!

115

A cost function (or loss function) is a mathematical tool that evaluates how well a machine learning model performs by measuring all its errors and outputting a single score. Lower scores indicate better model performance. Mean Squared Error (MSE) is the most commonly used cost function for regression problems. The calculation involves four steps: (1) Calculate the error by subtracting predicted from actual values, (2) Square each error to eliminate negative signs and penalize larger errors more heavily, (3) Sum all squared errors, and (4) Divide by the number of data points to get the average. This process transforms multiple prediction errors into a single numerical score that represents the model's overall accuracy.

Click to watch →

Deep Dive

Fundamental linear algebra, particularly matrix multiplication and vector representations of data.

Deep Learning Chapter 2 Linear Algebra presented by Gavin Crooks

Deep Learning Chapter 2 Linear Algebra presented by Gavin Crooks

49K

Matrix multiplication is the most important operation in linear algebra. For matrix-vector multiplication, each row of the matrix multiplies with the vector elements and sums to produce output elements. For matrix-matrix multiplication, the number of columns in the first matrix must equal the number of rows in the second matrix. The linearity property means multiplying a matrix by a sum of vectors equals multiplying by each vector separately. This is the fundamental property that gives linear algebra its name and power.

Click to watch →

On Deep Learning by Ian Goodfellow et al: Linear Algebra | Chapter 2

On Deep Learning by Ian Goodfellow et al: Linear Algebra | Chapter 2

1.3K

Linear algebra uses four fundamental data structures: scalars (single numbers, lowercase italic), vectors (1D arrays, lowercase bold), matrices (2D arrays, uppercase letters), and tensors (higher-dimensional arrays for 3D+ data). Matrix transpose flips elements around the diagonal, where (i,j) moves to (j,i). Matrix multiplication requires compatible dimensions: columns of first matrix must equal rows of second matrix. The result has dimensions M×P when multiplying M×N by N×P matrices. Dot product calculates sum of element-wise products between vectors. Matrix multiplication is performed by taking dot products between rows of the first matrix and columns of the second matrix. These operations form the foundation for linear transformations and are extensively used in deep learning for layer computations.

Click to watch →

Was ist eine Matrix? - Matrizen

Was ist eine Matrix? - Matrizen

761.8K

A matrix is a rectangular array of numbers organized into rows and columns, with dimensions specified as m x n indicating m rows and n columns. Matrices contain exactly m × n elements. Matrix-vector multiplication involves multiplying each row of the matrix by the corresponding vector components and summing the results to produce a new vector. This operation enables the representation of linear systems of equations in matrix form, where coefficients become matrix entries and constants form a separate vector. The matrix representation simplifies solving systems through row operations, making it equivalent to working with the tabular form used in Gaussian elimination algorithms.

Click to watch →

Ruben van de Geer: A Primer (or Refresher) On Linear Algebra for Data Science | PyData London 2019

Ruben van de Geer: A Primer (or Refresher) On Linear Algebra for Data Science | PyData London 2019

9.1K

Linear algebra deals with matrices and vectors, serving as the mathematical foundation for modern technology including Google's original search engine and data science applications. Vectors represent coordinates relative to the origin, while matrices are collections of column vectors that transform inputs to outputs. Matrix-vector multiplication computes weighted sums of columns. Geometrically, matrices perform transformations like rotation—applying a rotation matrix to each pixel vector rotates an entire image. The inverse matrix reverses transformations. Linear regression finds the best-fitting line by locating the parameter point closest to observed data, where the error vector must be orthogonal to the prediction plane. Two vectors are orthogonal if their dot product equals zero, and minimizing error norm achieves root mean square error minimization.

Click to watch →

선형대수학 무료강의 - [서울대 AI박사] - 행렬, 벡터, 스칼라, 텐서 등 인공기능 수학 기초

선형대수학 무료강의 - [서울대 AI박사] - 행렬, 벡터, 스칼라, 텐서 등 인공기능 수학 기초

46.2K

Linear algebra provides mathematical frameworks for representing and manipulating data essential for artificial intelligence. Scalars are single numerical values on a number line. Vectors are ordered collections of scalars arranged in one dimension, representing quantities with both magnitude and direction. Matrices are two-dimensional arrays of vectors, forming rectangular grids. Tensors generalize these concepts to higher dimensions. All real-world phenomena can be represented as mathematical data: images as pixel grids with RGB values, text as character sequences, and numerical values as scalars. The fundamental operations in linear algebra are addition/subtraction and scaling (multiplication by scalars). Linear independence determines whether vectors can span their entire space—two vectors in 2D are independent if they point in different directions. Basis vectors are linearly independent vectors that span an entire space—exactly n vectors are needed for n-dimensional space. Matrix rank measures the maximum number of linearly independent rows or columns in a matrix. A matrix with full rank spans its entire row or column space. Rank reduction occurs when vectors are linearly dependent. Matrix-vector multiplication follows dimensional rules: an m×n matrix multiplied by an n×1 vector produces an m×1 vector. The null space consists of all vectors x satisfying Ax = 0, revealing when matrices collapse dimensions.

Click to watch →

Backprop Basics

Parameter Setup

Network Outputs

Error Measurement

Gradient Descent

Chain Rule

Derivative Done

Optimizing b3

Key Takeaways

Backprop Basics

0:00

Playing Section

1
Introduces backpropagation as optimizing neural network weights and biases.
2
Recaps neural network structure and activation function transformations.
3
Plans to cover chain rule and gradient descent for parameter optimization.

Deep Dive

Advanced optimization techniques beyond basic gradient descent, such as Adam, RMSprop, and Momentum.

NNDL UNIT - 5|Optimization Strategies, Large-Scale Deep Learning,Computer Vision,NLP|JNTUH #r18 #r22

NNDL UNIT - 5|Optimization Strategies, Large-Scale Deep Learning,Computer Vision,NLP|JNTUH #r18 #r22

5.7K

Advanced optimizers address limitations of basic gradient descent. RMSprop (Root Mean Square Propagation) addresses rapidly diminishing learning rates by using a moving average of squared gradients to normalize updates, making it more stable for non-stationary settings. Adam (Adaptive Moment Estimation) combines momentum and RMSprop ideas, computing adaptive learning rates for each parameter using both first moment (mean) and second moment (variance) estimates of gradients. Adam maintains bias-corrected moment estimates and updates parameters using θ = θ - α * m̂ / (v̂ + ε). Advantages include efficiency and handling sparse gradients, while disadvantages include higher memory usage and sensitivity to hyperparameter choices. These optimizers are essential for training deep neural networks effectively.

Click to watch →

Lesson 5: Deep Learning 2019 - Back propagation; Accelerated SGD; Neural net from scratch

Lesson 5: Deep Learning 2019 - Back propagation; Accelerated SGD; Neural net from scratch

88.8K

Momentum maintains velocity from previous updates: velocity = momentum × velocity + learning_rate × gradient; parameters += velocity. This accelerates convergence in consistent gradient directions. RMSProp adapts learning rates based on gradient magnitudes: gradient = original_gradient / sqrt(rmsprop_term + ε). Adam combines both: step = momentum_term / sqrt(rmsprop_term) + ε. These adaptive methods handle different parameter scales and training dynamics. One-cycle learning rate policies dramatically accelerate training by increasing then decreasing learning rates strategically.

Click to watch →

Lecture 6 | NN Training, Regularization, Optimization | 27 February 2026 | CSE425

Lecture 6 | NN Training, Regularization, Optimization | 27 February 2026 | CSE425

415

This section covers advanced optimization techniques for training neural networks. Momentum adds a velocity term to gradient descent that accumulates past gradients, helping the optimizer move in the correct direction and escape local minima. The momentum term is a fraction (typically 0.9-0.99) of the previous velocity. Adaptive learning rate methods adjust the learning rate for each parameter based on its historical gradients. RMSprop maintains a moving average of squared gradients and divides the learning rate by this moving average. Adam (Adaptive Moment Estimation) combines the benefits of momentum and RMSprop by maintaining both first and second moment estimates of gradients, applying bias correction for early training, and updating parameters using both moment estimates. Adam is typically the default optimizer for deep learning because it combines the advantages of both techniques and is computationally efficient.

Click to watch →

CMU Introduction To Deep Learning 11-785, Fall 2025: Lecture 7

CMU Introduction To Deep Learning 11-785, Fall 2025: Lecture 7

563

This extensive section covers advanced optimization algorithms that improve upon basic gradient descent. It begins with mini-batch gradient descent, which balances computational efficiency and update stability by processing small randomly selected subsets of training data, with variance decreasing proportionally with batch size. The section then introduces momentum methods, which maintain running averages of past gradients to accumulate velocity in consistent directions while dampening oscillations. Nesterov accelerated gradient improves upon momentum by first applying the momentum step to predict the next position, then computing the gradient at this predicted location. RMSProp addresses oscillatory updates by maintaining running averages of squared derivatives per parameter, scaling learning rates adaptively to reduce steps in volatile directions. Finally, Adam combines momentum and RMSProp, maintaining separate running averages for mean and variance with learning rate proportional to inverse RMS. These adaptive methods eliminate manual learning rate scheduling but require careful hyperparameter tuning.

Click to watch →

Stanford CS231N | Spring 2025 | Lecture 3: Regularization and Optimization

Stanford CS231N | Spring 2025 | Lecture 3: Regularization and Optimization

36.1K

SGD faces fundamental challenges: poor conditioning causes oscillation in narrow valleys, saddle points trap optimization in high-dimensional spaces, and noisy updates slow convergence. Momentum addresses these by accumulating velocity from previous gradient directions, helping overcome oscillations and escape local minima. RMSProp normalizes gradient updates by dividing by running averages of squared gradients, adapting step sizes per-parameter based on historical behavior. Adam combines momentum and RMSProp with bias correction for initial steps. AdamW separates weight decay from optimization steps, preferred in practice for minimal hyperparameter tuning requirements.

Click to watch →

Deep Dive

The vanishing and exploding gradient problems, and techniques to mitigate them like batch normalization and proper weight initialization.

Tutorial-27:Vanishing gradient problem in neural networks |Deep Learning

Tutorial-27:Vanishing gradient problem in neural networks |Deep Learning

204

Five solutions address vanishing gradients: (1) Reduce network depth - smaller networks reduce multiplicative gradient shrinkage but compromise pattern capture; (2) Use ReLU activation - outputs zero for negatives and input for positives, preventing gradient shrinkage but risking 'dying ReLU'; (3) Batch normalization - normalizes layer inputs to reduce internal covariate shift and stabilize gradient flow; (4) Proper weight initialization - techniques like He or Xavier initialization ensure gradients remain within reasonable ranges; (5) Residual networks - introduce skip connections allowing gradients to flow directly through shortcuts. Additionally, the exploding gradient problem occurs when gradient values become excessively large, causing drastic weight changes and unstable training. Both problems result from improper gradient scaling during backpropagation in deep networks.

Click to watch →

Lecture 3 Back Propagation + Sheet 2 |Deep Learning

Lecture 3 Back Propagation + Sheet 2 |Deep Learning

401

The exploding gradient problem occurs when gradients become very large during backpropagation, causing weights to update by large amounts. This can cause the network to become unstable and fail to converge. Exploding gradients can happen when using activation functions like ReLU in deep networks, or when using very large learning rates. Solutions include gradient clipping, weight initialization techniques, and batch normalization. Batch normalization normalizes layer inputs to have zero mean and unit variance, helping prevent both vanishing and exploding gradients.

Click to watch →

[PyTorch] Lab-09-4 Batch Normalization

[PyTorch] Lab-09-4 Batch Normalization

8.4K

Neural networks face gradient vanishing (gradients too small, stopping learning) and gradient exploding (gradients too large, causing instability). Three indirect solutions exist: changing activation functions (e.g., ReLU), proper weight initialization, and small learning rates. Batch normalization directly addresses these problems by normalizing layer inputs using the formula: x_norm = (x - mean) / (std + epsilon), followed by learnable gamma (scale) and beta (shift) parameters. This normalization prevents internal covariate shift, where input distributions to each layer change during training, accumulating with deeper networks. The technique stabilizes training, improves learning speed, and reduces the need for other regularization techniques like dropout.

Click to watch →

Vanishing and exploding gradient problem in Hindi | Deep Learning

Vanishing and exploding gradient problem in Hindi | Deep Learning

9.5K

In deep learning, the vanishing gradient problem occurs when gradients become extremely small during backpropagation, preventing the network from learning effectively, while the exploding gradient problem happens when gradients become excessively large, causing unstable training; both issues arise from the chain rule application in deep networks and can be mitigated through techniques like weight initialization, batch normalization, and activation functions like ReLU.

Click to watch →

🔥 Artificial Neural Networks (ANN) in Telugu | Forward & Backpropagation | Deep Learning #2

🔥 Artificial Neural Networks (ANN) in Telugu | Forward & Backpropagation | Deep Learning #2

139

Batch normalization is a technique used to handle vanishing and exploding gradient problems. The goal is to bring all weights to the same scale (typically between 0 and 1) so they don't vary too much. By normalizing weights, we prevent weights from becoming zero or becoming very large. Batch normalization has two main benefits: it prevents gradient problems and enables faster processing because mathematical operations are easier when all values are on the same scale.

Click to watch →

Deep Dive

Implementing automatic differentiation and backpropagation programmatically using frameworks like PyTorch or TensorFlow.

PyTorch Tutorial 04 - Backpropagation - Theory With Example

PyTorch Tutorial 04 - Backpropagation - Theory With Example

142.2K

PyTorch simplifies backpropagation through automatic differentiation. To implement: (1) Create tensors with torch.tensor() and set requires_grad=True for parameters needing gradients, (2) Perform forward pass by computing predictions and loss, (3) Call loss.backward() to automatically compute local gradients and perform the backward pass, (4) Access gradients via the .grad attribute on parameters. This eliminates manual gradient computation while maintaining the same mathematical principles demonstrated in the numerical example.

Click to watch →

Building a Bigram Language Model (Like GPT!) with Andrej Karpathy | Part 2

Building a Bigram Language Model (Like GPT!) with Andrej Karpathy | Part 2

109

Automatic differentiation can be implemented manually by defining operations that track both values and gradients. Each operation creates a new value object that knows how to compute its derivative. During forward propagation, values are computed normally. During backward propagation, gradients are computed by applying the chain rule in reverse order. This manual implementation demonstrates the underlying mechanics of frameworks like PyTorch and micrograd, showing how gradients flow through computational graphs to enable gradient-based optimization.

Click to watch →

Introduction to Deep Learning - Module 1 - Video 19: Computational Complexity/Approaches of BP

Introduction to Deep Learning - Module 1 - Video 19: Computational Complexity/Approaches of BP

102

Two main approaches exist for implementing backpropagation: symbol-to-number differentiation (numerical gradient computation used in PyTorch/Caffe) and symbol-to-symbol differentiation (symbolic graph construction used in TensorFlow/Theano). The latter allows modular extension to higher-order derivatives and general tensor operations, making it suitable for broader computational tasks beyond neural networks.

Click to watch →

Введение в глубокое обучение 1. Введение в глубокое обучение

Введение в глубокое обучение 1. Введение в глубокое обучение

913

Automatic differentiation frameworks (like PyTorch) implement backpropagation by defining forward and backward operations for each computational block. Users compose blocks into computational graphs, and the framework automatically computes gradients. This enables efficient training of complex models without manual gradient derivation. PyTorch is essentially an automatic differentiation framework where neural networks are the primary application.

Click to watch →

PyTorch. Основы нейросети. Теория и практика для начинающих

PyTorch. Основы нейросети. Теория и практика для начинающих

12.6K

Backpropagation computes gradients by traversing the computational graph in reverse (from output to input) and applying the chain rule. Different operations have different gradient propagation rules: addition passes gradients unchanged, multiplication multiplies by the other operand, and sine multiplies by cosine of the input. For branching paths, gradients accumulate: if z = f(g(x) + h(x)), then dz/dx = (df/dg) × (dg/dx) + (df/dh) × (dh/dx). PyTorch uses automatic differentiation to compute gradients. When you create tensors with requires_grad=True, PyTorch automatically builds a computational graph that tracks all operations. Calling .backward() on the output tensor computes gradients for all parameters. This is more flexible than symbolic differentiation (like TensorFlow's static graphs) and more efficient than numerical differentiation. PyTorch uses dynamic computational graphs (define-by-run), while TensorFlow uses static graphs (define-before-run).

Click to watch →

Deep Dive

Regularization methods (e.g., L1/L2 regularization and dropout) to prevent overfitting during neural network training.

Week 2 – Lecture: Stochastic gradient descent and backpropagation

Week 2 – Lecture: Stochastic gradient descent and backpropagation

58.2K

Regularization prevents overfitting by adding penalties to the loss function. L2 regularization (weight decay) adds λ||W||², shrinking weights toward zero proportionally: W_new = W_old × (1-αλ) - α∇L. L1 regularization adds λ∑|W|, driving some weights exactly to zero and creating sparse representations. Dropout randomly sets neuron outputs to zero during training (typically 50%), forcing distributed information encoding. During testing, outputs are scaled by dropout rate. Proper weight initialization (He initialization: variance 1/n_in) ensures stable signal propagation through deep networks, preventing vanishing/exploding gradients.

Click to watch →

Regularization in Neural Networks

Regularization in Neural Networks

542

Regularization prevents overfitting in neural networks by adding penalty terms to the objective function (L1/L2 regularization) or by randomly removing neurons during training (dropout), where dropout scales the outbound weights of each neuron by the probability that the neuron was present during training.

Click to watch →

Técnicas de regularización para redes neuronales

Técnicas de regularización para redes neuronales

1.1K

Regularization techniques prevent overfitting by constraining network complexity. L2 regularization adds a penalty term to the loss function: the sum of squared weights multiplied by a hyperparameter λ. This discourages large weights and encourages all weights to be small and distributed evenly. L1 regularization uses absolute values of weights instead of squared weights, causing weights to become exactly zero (feature selection). Dropout randomly deactivates neurons during training with a certain probability (typically 0.1 to 0.5), preventing neurons from becoming too specialized and dependent on each other. During validation, dropout is not used, and all neurons are active. To compensate for reduced signal during training, outputs are scaled by dividing by the dropout probability. These techniques make training more difficult but improve generalization performance.

Click to watch →

[AM - Aula 6 Tópico 2] Use de Dropout e Regularização L1/L2 para redução do Overfitting

[AM - Aula 6 Tópico 2] Use de Dropout e Regularização L1/L2 para redução do Overfitting

363

In deep neural networks, the number of parameters (weights) between layers can become extremely large (e.g., 525,600 weights in a network with 1000 input features, 500 neurons in the first hidden layer, and 50 neurons in the second layer), which increases model complexity and the risk of overfitting. To prevent overfitting, two main regularization techniques are used: (1) Weight regularization (L1 and L2) adds a penalty term to the loss function based on the magnitude of weights, where L2 regularization (Ridge) penalizes large weights by adding the squared norm of weights, and L1 regularization (Lasso) can drive some weights to zero, effectively performing feature selection; (2) Dropout randomly disables a percentage of neurons during training (e.g., 25%), reducing the effective number of parameters and preventing co-adaptation of neurons. The regularization parameter λ controls the trade-off between fitting the training data and maintaining generalization performance.

Click to watch →

Lecture 11 Training NN |Deep Learning

Lecture 11 Training NN |Deep Learning

220

Regularization is the primary solution to overfitting in neural networks, working by adding penalty terms to the loss function that discourage large weights. L1 regularization adds a penalty proportional to the absolute value of weights (|w|), producing sparse solutions with many weights exactly zero. L2 regularization adds a penalty proportional to the square of weights (w²), producing solutions with small but non-zero weights. Dropout randomly 'drops' a fraction of neurons during training, preventing the model from becoming too dependent on any single neuron and forcing it to learn more robust features. Dropout can be viewed as an ensemble method where each training iteration creates a different subnetwork. These techniques help models generalize better by preventing them from memorizing the training data.

Click to watch →

Backprop Basics

Parameter Setup

Network Outputs

Error Measurement

Gradient Descent

Chain Rule

Derivative Done

Optimizing b3

Key Takeaways

Backprop Basics

0:00

Playing Section

1
Introduces backpropagation as optimizing neural network weights and biases.
2
Recaps neural network structure and activation function transformations.
3
Plans to cover chain rule and gradient descent for parameter optimization.

732K views14.4K17:33@statquestOriginal Release: 2020-10-19

Backpropagation optimizes neural network parameters by using the chain rule to calculate derivatives of the loss function (sum of squared residuals) with respect to each parameter, then applying gradient descent to iteratively adjust parameters toward optimal values; this process starts from the last parameter and works backward through the network.

[00:00:00]Backpropagation is a really big word, but it's not a really big deal. StatQuest!

[00:00:08]Hello!

[00:00:09]I'm Josh Starmer and welcome to StatQuest!

[00:00:11]Today we're going to talk about Neural Networks, Part 2: Backpropagation Main Ideas.

[00:00:19]Note: this StatQuest assumes that you are already familiar with neural networks, the chain rule, and gradient descent. If not, check out the quests. The links are in the description below.

[00:00:35]In the StatQuest on Neural Networks Part 1, Inside the Black Box, we started with a simple dataset that showed whether or not different drug dosages were effective against a virus.

[00:00:49]The low and high dosages were not effective, but the medium dosage was effective.

[00:00:57]Then we talked about how a neural network like this one fits a green squiggle to this data set.

[00:01:05]Remember, the neural network starts with identical activation functions, but, using different weights and biases on the connections, it flips and stretches the activation functions into new shapes, which are then added together to get a squiggle that is shifted to fit the data.

[00:01:27]However, we did not talk about how to estimate the weights and biases.

[00:01:33]So let's talk about how backpropagation optimizes the weights and biases in this, and other, neural networks.

[00:01:42]Note: backpropagation is relatively simple, but there are a ton of details, so I split it up into bite-sized pieces.

[00:01:52]In this part we talk about the main ideas of backpropagation.

[00:01:58]One: using the chain rule to calculate derivatives, and Two: plugging the derivatives into gradient descent to optimize parameters. In the next part we'll talk about how the chain rule and gradient descent apply to multiple parameters simultaneously, and introduce some fancy notation.

[00:02:21]Then we will go completely bonkers with the chain rule and show how to optimize all seven parameters simultaneously in this neural network.

[00:02:31]Bam!

[00:02:34]First, so we can be clear about which specific weights we are talking about, let's give each one a name: we have w1, w2, w3, and w4.

[00:02:51]And let's name each bias: b1, b2, and b3.

[00:03:00]Note: conceptually, backpropagation starts with the last parameter and works its way backwards to estimate all of the other parameters.

[00:03:11]However, we can discuss all of the main ideas behind a backpropagation by just estimating the last bias, b3.

[00:03:21]So, in order to start from the back, let's assume that we already have optimal values for all of the parameters except for the last bias term, b3.

[00:03:34]Note: throughout this, and the next StatQuests, I'll make the parameter values that have already been optimized green, and unoptimized parameters will be red.

[00:03:47]Also, note: to keep the math simple, let's assume dosages go from 0, for low, to 1, for high.

[00:03:56]Now, if we run dosages from 0 to 1 through the connection to the top node in the hidden layer, then we get the x-axis coordinates for the activation function, that are all inside this red box and when we plug the x-axis coordinates into the activation function which, in this example, is the soft plus activation function, we get the corresponding y-axis coordinates, and this blue curve.

[00:04:30]Then we multiply the y-axis coordinates on the blue curve by negative 1.22 and we get the final blue curve.

[00:04:40]Bam!

[00:04:42]Now, if we run dosages from zero to one through the connection to the bottom node in the hidden layer, then we get x-axis coordinates inside this red box.

[00:04:56]Now we plug those x-axis coordinates into the activation function to get the corresponding y-axis coordinates for this orange curve.

[00:05:08]Now we multiply the y-axis coordinates on the orange curve by negative 2.3 and we end up with this final orange curve.

[00:05:19]Bam!

[00:05:22]Now we add the blue and orange curves together to get this green squiggle.

[00:05:29]Now we are ready to add the final bias, b3, to the green squiggle.

[00:05:35]Because we don't yet know the optimal value for b3, we have to give it an initial value, and because bias terms are frequently initialized to 0, we will set b3 equal to 0.

[00:05:50]Now, adding zero to all of the y-axis coordinates on the green squiggle leaves it right where it is.

[00:05:58]However, that means the green squiggle is pretty far from the data that we observed.

[00:06:05]We can quantify how good the green squiggle fits the data by calculating the sum of the squared residuals.

[00:06:12]A residual is the difference between the observed and predicted values.

[00:06:18]For example, this residual is the observed value, zero, minus the predicted value from the green squiggle, negative 2.6. This residual is the observed value, one, minus the predicted value from the green squiggle, negative 1.61.

[00:06:41]Lastly, this residual is the observed value, 0, minus the predicted value from the green squiggle, negative 2.61.

[00:06:52]Now we square each residual and add them all together to get 20.4 for the sum of the squared residuals.

[00:07:03]So when b3 equals 0, the sum of the squared residuals equals 20.4. And that corresponds to this location on this graph that has the sum of the squared residuals on the y -axis and the bias, b3, on the x-axis.

[00:07:24]Now, if we increase b3 to 1, then we would add one to the y-axis coordinates on the green squiggle and shift the green squiggle up one.

[00:07:38]And we end up with shorter residuals.

[00:07:42]When we do the math, the sum of the squared residuals equals 7.8, and that corresponds to this point on our graph.

[00:07:55]If we increase b3 to 2, then the sum of the squared residuals equals 1.11. And if we increase b3 to 3, then the sum of the squared residuals equals 0.46. And if we had time to plug in tons of values for b3, we would get this pink curve, and we could find the lowest point, which corresponds to the value for b3 that results in the lowest sum of the squared residuals, here.

[00:08:28]However, instead of plugging in tons of values to find the lowest point in the pink curve, we use gradient descent to find it relatively quickly.

[00:08:39]And that means we need to find the derivative of the sum of the squared residuals with respect to b3.

[00:08:47]Now, remember the sum of the squared residuals equals the first residual squared, plus all of the other squared residuals.

[00:08:58]Now, because this equation takes up a lot of space, we can make it smaller by using summation notation.

[00:09:08]The greek symbol sigma tells us to sum things together, and 'i' is an index for the observed and predicted values that starts at one.

[00:09:20]And the index goes from one to the number of values, 'n', which in this case is set to 3.

[00:09:28]So, when 'i' equals one, we're talking about the first residual.

[00:09:34]When 'i' equals two, we're talking about the second residual.

[00:09:39]And when 'i' equals three, we are talking about the third residual.

[00:09:44]Now let's talk a little bit more about the predicted values.

[00:09:49]Each predicted value comes from the green squiggle, and the green squiggle comes from the last part of the neural network.

[00:09:59]In other words, the green squiggle is the sum of the blue and orange curves, plus b3.

[00:10:08]Now remember, we want to use gradient descent to optimize b3, and that means we need to take the derivative of the sum of the squared residuals with respect to b3.

[00:10:22]And because the sum of the squared residuals are linked to b3 by the predicted values, we can use the chain rule to solve for the derivative of the sum of the squared residuals with respect to b3.

[00:10:41]The chain rule says that the derivative of the sum of the squared residuals with respect to b3 is the derivative of the sum of the squared residuals with respect to the predicted values, times the derivative of the predicted values with respect to b3.

[00:10:59]Now, before we calculate the derivative of the sum of the squared residuals with respect to the predicted values, let's clean up our workspace and move these equations out of the way.

[00:11:13]Now we can solve for the derivative of the sum of the squared residuals with respect to the predicted values by first substituting in the equation, and then use the chain rule to move the square to the front, and then we multiply that by the derivative of the stuff inside the parentheses with respect to the predicted values, negative one.

[00:11:37]Now we simplify by multiplying two by negative 1, and we have the derivative of the sum of the squared residuals with respect to the predicted values.

[00:11:49]So let's move that up here, and now we are done with the first part.

[00:11:55]Now let's solve for the second part: the derivative of the predicted values with respect to b3.

[00:12:03]We start by plugging in the equation for the predicted values.

[00:12:08]Remember, the blue and orange curves were created before we got to b3.

[00:12:14]So the derivative of the blue curve with respect to b3 is 0, because the blue curve is independent of b3.

[00:12:24]And the derivative of the orange curve with respect to b3 is also 0.

[00:12:30]Lastly, the derivative of b3, with respect to b3, is 1.

[00:12:36]Now we just add everything up, and the derivative of the predicted values with respect to b3, is one.

[00:12:46]So we multiply the derivative of the sum of the squared residuals with respect to the predicted values by 1.

[00:12:54]Note: this times 1 part in the equation doesn't do anything, but I'm leaving it in to remind us that the derivative of the sum of the squared residuals with respect to b3 consists of two parts: the derivative of the sum of the squared residuals with respect to the predicted values, and the derivative of the predicted values with respect to b3.

[00:13:20]Bam! And at long last we have the derivative of the sum of the squared residuals with respect to b3.

[00:13:30]And that means we can plug this derivative into gradient descent to find the optimal value for b3.

[00:13:38]So let's move this equation up and show how we can use this equation with gradient descent.

[00:13:45]Note: if you're not familiar with gradient descent, check out the quest the link is in the description below.

[00:13:53]Anyway, first, we expand the summation.

[00:13:58]Then, we plug in the observed values and the values predicted by the green squiggle.

[00:14:05]Remember, we get the predicted values on the green squiggle by running the dosages through the neural network.

[00:14:13]Now, we just do the math and get negative 15.7. And that corresponds to the slope for when b3 equals zero.

[00:14:25]Now we plug the slope into the gradient descent equation for step size, and, in this example, we'll set the learning rate to 0.1. And that means the step size is -1.57.

[00:14:41]Now we use the step size to calculate the new value for b3 by plugging in the current value for b3, zero, and the step size, -1.57. And the new value for b3 is 1.57.

[00:15:00]Changing b3 to 1.57 shifts the green squiggle up, and that shrinks the residuals.

[00:15:09]Now, plugging in the new predicted values and doing the math gives us -6.26, which corresponds to the slope when b3 equals 1.57.

[00:15:23]Then, we calculate the step size and the new value for b3, which is 2.19.

[00:15:34]Changing b3 to 2.19 shifts the green squiggle up further, and that shrinks the residuals even more.

[00:15:44]Now we just keep taking steps until the step size is close to zero. And because the step size is close to 0 when b3 equals 2.61, we decide that 2.61 is the optimal value for b3.

[00:16:03]Double bam!

[00:16:06]So, the main ideas for backpropagation are that, when a parameter is unknown, like b3, we use the chain rule to calculate the derivative of the sum of the squared residuals with respect to the unknown parameter, which in this case was b3.

[00:16:25]Then we initialize the unknown parameter with a number, and in this case we set b3 equal to zero, and used gradient descent to optimize the unknown parameter.

[00:16:39]Triple bam!

[00:16:42]In the next StatQuest we'll show how these ideas can be used to optimize all of the parameters in a neural network.

[00:16:50]Now it's time for some shameless self-promotion.

[00:16:56]If you want to review statistics and machine learning offline, check out the StatQuest study guides at statquest.org.

[00:17:04]There's something for everyone.

[00:17:06]Hooray!

[00:17:07]We've made it to the end of another exciting StatQuest.

[00:17:11]If you like this StatQuest and want to see more, please subscribe. And if you want to support StatQuest, consider contributing to my patreon campaign, becoming a channel member, buying one or two of my original songs, or a t-shirt or a hoodie, or just donate the links are in the description below. Alright, until next time. Quest on!

Up Next

Physics-Informed Neural Networks: PINN Tutorial with Drug Decay Example

@zara-dar

70.5K views•2025-12-13

Physics-Informed Deep Learning: PINNs Tutorial (CAII HAL Training)

@NCSAatIllinois

2.1K views•2021-11-19

Decision Trees and Classification Trees Explained (Machine Learning Basics)

@statquest

1.1M views•2021-04-26

Neural Networks Explained: Math, Layers, and Learning Fundamentals

@3blue1brown

21.9M views•2017-10-05

Related Study Plans & Knowledge Roadmaps

Structured learning paths in Artificial Intelligence

View All Artificial Intelligence Topics

artificial-intelligence

PINNs: Fluid Dynamics with PyTorch & DeepXDE

01Calculus & Fluid Dynamics Foundations

02Deep Learning & PyTorch Basics

03Introduction to Physics-Informed Neural Networks (PINNs)

+ 3 more learning modules

Explore Roadmap

artificial-intelligence

Aligning LLMs: DPO, TRL & Custom Datasets

01Foundations of Python & Machine Learning

02Transformers and Large Language Models (LLMs)

03Aligning LLMs: SFT and RLHF

+ 3 more learning modules

Explore Roadmap

artificial-intelligence

Federated Learning: PyTorch, Flower & Privacy

01Python & Machine Learning Foundations

02Deep Learning Basics with PyTorch

03Introduction to Federated Learning

+ 2 more learning modules

Explore Roadmap

artificial-intelligence

Securing LLMs: Guardrails AI & NeMo Gateway

01Introduction to LLMs, Python, and APIs

02Understanding LLM Vulnerabilities & Threats

03Defending LLMs with Guardrails AI

+ 2 more learning modules

Explore Roadmap