Academic Databases
Explore Our Semantic Databases
STEM & Tech
Math, Physics, CS & Engineering...
Economics
Macro, Micro, Econometrics...
Business
Management, Marketing, Finance...
Health & Life Sciences
Biology, Medicine, Neuroscience...
Humanities & Social Sciences
Psychology, Law, Philosophy, History...
General Studies
General & interdisciplinary studies...
Explore Topics
Features

Linear Regression Gradient Descent Explained | Machine Learning Tutorial

Added: 2026-08-07

Deep Dive

Understanding Simple Linear Regression, including the equation of a line (y = mx + b) and the roles of the slope and intercept.

Machine Learning Full Course 2026 | Learn Machine Learning | Machine Learning Tutorial | Simplilearn

Machine Learning Full Course 2026 | Learn Machine Learning | Machine Learning Tutorial | Simplilearn

88.5K

The simple linear regression equation is y = mx + c, where y is the dependent variable, x is the independent variable, m is the slope (change in y per unit change in x), and c is the y-intercept (value of y when x is zero). The slope is calculated as (y2 - y1)/(x2 - x1), representing the rate of change between variables.

Click to watch →

5.1 Exploring the Equation of a line

5.1 Exploring the Equation of a line

561

The equation Y = mx + b represents a line where 'm' is the slope (steepness and direction) and 'b' is the y-intercept (where the line crosses the y-axis); positive slope means the line rises to the right, negative slope means it falls, horizontal lines have slope = 0, and vertical lines have undefined slope.

Click to watch →

Slope-Intercept Form y=mx+b

Slope-Intercept Form y=mx+b

2.3M

The slope-intercept formula is y = mx + b, where m represents the slope of a line and b represents the y-intercept. The formula provides everything needed to plot a line on a coordinate plane. Any x and y values that satisfy this equation represent coordinates on that specific line.

Click to watch →

Linear Relations Top 10 Must Knows (ultimate study guide)

Linear Relations Top 10 Must Knows (ultimate study guide)

62.2K

The slope-intercept form y = mx + b is the most common representation of linear relationships, where m represents the slope (rate of change) and b represents the y-intercept (where the line crosses the y-axis). To derive this equation from a graph: first identify the y-intercept (b-value) by finding where the line crosses the y-axis, then calculate the slope (m-value) by counting rise over run between two known points. To graph from the equation: plot the y-intercept point, then use the slope as a fraction (rise/run) to find additional points - for example, a slope of -2 can be interpreted as down 2, right 1 or up 2, left 1.

Click to watch →

Algebra Basics: Inequalities In Algebra - Math Antics

Algebra Basics: Inequalities In Algebra - Math Antics

753K

The equation y = mx + b represents a straight line in the coordinate plane, where m is the slope (steepness) and b is the y-intercept (where the line crosses the y-axis). When m = 1 and b = 0, this simplifies to y = x, which is a special case where the line passes through the origin at a 45-degree angle, meaning every ordered pair solution has the same x and y values.

Click to watch →

Deep Dive

Basic Calculus concepts, particularly derivatives and partial derivatives, to understand how to calculate rates of change and slopes.

2.2a Basic Differentiation Rules and Rates of Change - Calculus

2.2a Basic Differentiation Rules and Rates of Change - Calculus

20.6K

A derivative represents both the rate of change and the slope of a function at a given point. The derivative of any constant function f(x) = c is zero because a horizontal line has no slope. The power rule states that for f(x) = x^n, the derivative f'(x) = n*x^(n-1). For linear functions f(x) = x, the derivative is 1. These foundational rules allow calculation of instantaneous rates of change and slopes at specific points on a graph.

Click to watch →

(L-09) Calculus, Derivative, Partial derivative, Del Operator | BSC Physics Semester 1 | LNMU Notes

(L-09) Calculus, Derivative, Partial derivative, Del Operator | BSC Physics Semester 1 | LNMU Notes

243

Calculus is a branch of physics that studies change—how things change, how fast they change, or how slow they change. The fundamental principle is that change is the law of nature, and calculus provides the mathematical tools to analyze these changes. Calculus has two main ideas: differentiation (finding rates of change) and integration (accumulation). A derivative is the rate of change of a quantity, measuring how quickly something is transforming. For any change to be described, two points are always required—a single point cannot be described as changing; change can only be described relative to another point. The rate of change is defined as the change in one quantity divided by the change in another quantity. Different fields use different terms for rate of change: in cricket, it's 'run rate'; in internet downloads, it's 'speed' (e.g., 30 MBPS means 30 megabytes per second). The derivative is specifically the instantaneous rate of change, measuring the rate at a particular instant rather than over an extended period. Differentiation involves three steps: (1) Choose a point where you want to find the derivative, (2) Choose another point extremely close to the first point, (3) Calculate the average rate of change between these two points. The interval between points must be infinitesimally small (approaching zero but not zero). The derivative formula is df/dx, representing the instantaneous rate of change of f with respect to x. The derivative represents the slope of the tangent line to the function's graph at a specific point. Partial derivatives are used when a function depends on multiple variables. To find a partial derivative with respect to one variable (say x), treat all other variables (like t) as constants. Partial derivatives are denoted by ∂ (partial derivative symbol) or sometimes by 'd' with context. An ordinary derivative (d/dx) is used for single-variable functions, while a partial derivative (∂/∂x) is used for multi-variable functions.

Click to watch →

Your First Basic CALCULUS Problem Let’s Do It Together….

Your First Basic CALCULUS Problem Let’s Do It Together….

582.7K

The derivative is a fundamental concept in calculus that gives the slope of a function at any specific point. Unlike linear equations which have constant slopes, curved functions have changing slopes that vary along their length. The derivative provides a formula that calculates this instantaneous slope at any given x-value. This concept is essential for understanding how quantities change and for finding maximum or minimum values in various applications.

Click to watch →

Basic Calculus - Derivatives of Algebraic and Exponential Functions

Basic Calculus - Derivatives of Algebraic and Exponential Functions

1.1K

A derivative is obtained through differentiation, the process of finding the rate of change. The study of differentiation is called differential calculus. Derivative, slope, and rate of change are equivalent concepts used in different contexts: derivative in equations, slope in graphs, and rate of change in worded problems. Notations include y', dy/dx, and f'(x). Slope at a point is represented by tangent lines, which can be positive (leans right), negative (leans left), or zero (horizontal). Rate of change describes how fast or slow something changes in applied problems.

Click to watch →

Hướng dẫn trọn bộ: CS231n - Deep Learning for Computer Vision. Note: Backpropagation

Hướng dẫn trọn bộ: CS231n - Deep Learning for Computer Vision. Note: Backpropagation

1.2K

The derivative of a function at a point represents the slope of the tangent line to the function's graph at that point. A steep slope (large derivative) means the function changes rapidly with small changes in x. A shallow slope (small derivative) means the function changes slowly. For a function f(x,y) = x² + y², the partial derivative with respect to x is 2x (treating y as constant), and the partial derivative with respect to y is 2y (treating x as constant). Partial derivatives measure how the function changes when only one variable changes, holding others constant. The notation ∂f/∂x (read as 'partial f with respect to x') denotes the partial derivative.

Click to watch →

Cost Function & GD Intro

GD Step Mechanics

Slope Direction Logic

Data & Matrix Setup

Prediction Formula

Vectorized Cost Derivative

GD Pseudo-code

Learning Rate Effects

Key Takeaway

Cost Function & GD Intro

0:00

Playing Section

1
Linear regression aims to fit a best-fit line by minimizing error.
2
Cost function measures prediction error; smaller is better.
3
Gradient descent is the algorithm used to minimize this cost.

Deep Dive

Exploring variations of Gradient Descent, such as Stochastic Gradient Descent (SGD) and Mini-batch Gradient Descent.

MIT Introduction to Deep Learning | 6.S191

MIT Introduction to Deep Learning | 6.S191

909.4K

Standard gradient descent computes gradients over entire datasets, which is computationally expensive for large problems. Stochastic Gradient Descent addresses this by computing gradients over single randomly selected data points, making training much faster though noisier. Mini-batch Gradient Descent strikes a balance by computing gradients over small batches of k data points (typically 32-128). This provides more reliable gradient estimates than single-sample methods while maintaining efficiency. The increased stability allows larger learning rates and enables parallelization across GPU cores, significantly accelerating training for practical deep learning applications.

Click to watch →

Stochastic Gradient Descent vs Batch Gradient Descent vs Mini Batch Gradient Descent |DL Tutorial 14

Stochastic Gradient Descent vs Batch Gradient Descent vs Mini Batch Gradient Descent |DL Tutorial 14

217.6K

This tutorial explains three variants of gradient descent optimization algorithms used in machine learning: Batch Gradient Descent computes the gradient using all training samples before updating weights, making it computationally expensive for large datasets but producing smooth convergence; Stochastic Gradient Descent updates weights after each individual training sample, enabling faster computation and handling large datasets but resulting in a zigzag convergence path; Mini-Batch Gradient Descent strikes a balance by using small random batches of samples, combining computational efficiency with smoother convergence than stochastic descent. The video demonstrates these concepts through a Python implementation of house price prediction using the housing dataset, showing how each algorithm affects the cost versus epoch graph and model performance.

Click to watch →

Tutorial 12- Stochastic Gradient Descent vs Gradient Descent

Tutorial 12- Stochastic Gradient Descent vs Gradient Descent

267K

Stochastic Gradient Descent (SGD) differs from standard Gradient Descent in how they compute weight updates during optimization: Gradient Descent uses the entire dataset to calculate the exact gradient (W_new = W_old - learning_rate × ∂L/∂W), requiring high computational resources for large datasets; SGD uses only one data point at a time, making it computationally efficient but introducing noise that causes zigzag convergence paths; Mini-Batch SGD strikes a balance by using a small subset (k < n) of data points per update, reducing computational load while maintaining reasonable convergence stability, which is why it's widely used in modern neural networks like CNNs.

Click to watch →

1.9 | Deep Learning | Gradient Descent & Types KCS-078 | AKTU & Other Universities

1.9 | Deep Learning | Gradient Descent & Types KCS-078 | AKTU & Other Universities

45.5K

Stochastic Gradient Descent (SGD) randomly selects one data point at each iteration, calculates its error, and immediately updates weights. This is computationally efficient for large datasets but produces noisy updates. Mini-Batch Gradient Descent combines Batch and Stochastic approaches by dividing data into small batches (e.g., 20 samples), calculating errors for each batch, computing the average, and updating weights once per batch. Mini-batch size is a hyperparameter requiring tuning. This approach balances computational efficiency with update stability, making it the most commonly used method in practice.

Click to watch →

Lecture 2 | Machine Learning (Stanford)

Lecture 2 | Machine Learning (Stanford)

846.8K

Stochastic gradient descent (SGD) is an alternative to batch gradient descent that updates parameters using only one randomly selected training example at each iteration. The update rule is θ_j := θ_j - α × (h_θ(x^(i)) - y^(i)) × x_j^(i). Unlike batch gradient descent which requires scanning the entire training set before each update, SGD can start learning immediately after seeing the first example. While it doesn't converge exactly to the minimum, it tends to oscillate around the minimum and often converges faster for large datasets.

Click to watch →

Deep Dive

Hyperparameter tuning, focusing on how to select and adjust the learning rate to prevent divergence or slow convergence.

Webinar: How do Machines "learn"? | Accolite Digital Connect

Webinar: How do Machines "learn"? | Accolite Digital Connect

137

The learning rate (alpha) determines step size in gradient descent. Too high causes divergence; too low causes slow training. Optimal learning rate selection is part of hyperparameter tuning. The machine learning training process follows a consistent pattern: define the model, define the cost function, use labeled training data, and apply an optimization algorithm. Algorithm selection depends on the problem: classification for categorical outputs, regression for quantitative predictions. Single input variables suggest linear regression, while multiple variables may require dimensionality reduction before applying multiple linear regression.

Click to watch →

Deep Learning(CS7015): Lec 5.7 Tips for Adjusting Learning Rate and Momentum

Deep Learning(CS7015): Lec 5.7 Tips for Adjusting Learning Rate and Momentum

34.3K

The learning rate (eta) controls step size in gradient descent. Large etas cause overshooting on gentle slopes and oscillations on steep regions. Learning rate is a hyperparameter, not learned from data. Effective tuning strategy: try values on log scale, run experiments for few epochs, select value with maximum loss decrease, then refine with linear search around optimal point. This addresses the wide range of possible learning rates (e.g., 0.0001 to 0.1).

Click to watch →

El Perceptrón - Descenso por Gradiente

El Perceptrón - Descenso por Gradiente

5.5K

Learning rate selection is critical: small rates cause slow convergence, large rates cause overshooting and oscillation, and excessively high rates cause divergence where loss increases instead of decreasing. The optimal learning rate depends on the specific loss function and model. Practitioners test multiple learning rates and select the one achieving the optimum in the fewest iterations.

Click to watch →

Основы глубинного обучения — семинар 4 (ИАД-9)

Основы глубинного обучения — семинар 4 (ИАД-9)

188

Learning rate is the most critical hyperparameter in neural network training. Large learning rates cause overshooting, divergence, or oscillation around the optimum. Small learning rates cause slow convergence and excessive computation. The example demonstrates that increasing LR from 0.001 to 0.01 reduced iterations from 729 to 3, but further increases cause divergence. Finding the right LR requires careful tuning and is often more important than model architecture.

Click to watch →

Lesson 3: Practical Deep Learning for Coders 2022

Lesson 3: Practical Deep Learning for Coders 2022

149.8K

The learning rate is a hyperparameter that controls step size during gradient descent. It multiplies the gradient when updating parameters. Too large and the optimizer overshoots the minimum and diverges; too small and convergence becomes very slow. The learning rate represents the balance between training speed and stability. Finding a good learning rate is crucial for successful model training.

Click to watch →

Deep Dive

Multiple Linear Regression, extending the gradient descent algorithm to handle datasets with multiple input features using vectorization.

Multiple Linear Regression with Gradient Descent from Scratch

Multiple Linear Regression with Gradient Descent from Scratch

2.2K

Vectorized gradient descent extends simple linear regression to handle multiple features efficiently. The key innovation is computing the gradient vector simultaneously using ∇J(θ) = (2/m) × X^T × (Xθ - y), producing a gradient vector where each element corresponds to a parameter's partial derivative. This approach eliminates the need for separate calculations for each parameter, dramatically improving computational efficiency. The implementation includes feature scaling using StandardScaler to normalize input ranges, adding a bias column to the input matrix, and demonstrating how the algorithm converges on complex real-world datasets like California housing data with multiple correlated features.

Click to watch →

tinytorch series Episode (3) Multiple Linear Regression.

tinytorch series Episode (3) Multiple Linear Regression.

200

Multiple Linear Regression extends single-parameter models by incorporating multiple input features (x1, x2, ..., xn) to predict a target variable (y), using the equation y = w0 + w1*x1 + w2*x2 + ... + wn*xn, where w0 is the bias and w1 through wn are the weights. The model is optimized using gradient descent, which computes partial derivatives of the cost function (mean squared error) with respect to each weight and updates them simultaneously. Vectorization transforms these operations into matrix operations (dot products), significantly improving computational efficiency by replacing explicit loops with optimized linear algebra operations. The number of parameters equals the number of input features plus one (for the bias term).

Click to watch →

Machine learning in C was a mistake

Machine learning in C was a mistake

94.2K

Multiple linear regression extends simple linear regression to handle multiple input features instead of just one. The model takes the form y = w₁x₁ + w₂x₂ + ... + wₙxₙ + b, where each wᵢ represents the weight for the corresponding feature xᵢ. The gradient descent algorithm can be generalized by treating each weight independently and updating them using the same update rule. This requires looping over all features to compute gradients and update weights, making the implementation scalable to any number of input dimensions.

Click to watch →

Gradient Descent Machine Learning | Gradient Descent Algorithm | Stochastic Gradient Descent Edureka

Gradient Descent Machine Learning | Gradient Descent Algorithm | Stochastic Gradient Descent Edureka

31.6K

Multiple linear regression extends simple regression by including multiple independent variables (e.g., area and bedrooms predicting price). Add an intercept column to the feature matrix for convenient matrix representation. Convert all variables to arrays. Use the same gradient descent framework but extend parameter vector θ to include coefficients for each feature plus the intercept. The update equations generalize to: θ_new = θ_old - learning_rate × (X^T × (X×θ - y)). After thousands of iterations, parameters converge to minimize the multi-dimensional cost surface. Plotting cost versus iterations reveals convergence behavior, typically flattening around 200 iterations with appropriate learning rates.

Click to watch →

Multiple Linear Regression | الانحدار الخطي المتعدد (بالعربي)

Multiple Linear Regression | الانحدار الخطي المتعدد (بالعربي)

190

Multiple Linear Regression extends simple linear regression to handle multiple independent variables (features) by using the equation y = ω₁x₁ + ω₂x₂ + ... + ωₙxₙ + ω₀, where ω₀ is the intercept. Vectorization transforms scalar operations into vector operations, enabling parallel computation that significantly improves efficiency (e.g., reducing computation time from 72ms to 2ms for 1000 features). The normal equation provides a closed-form solution ω = (X^T X)^(-1) X^T y, while gradient descent offers an iterative approach. Feature scaling is essential for gradient descent to converge efficiently, as it prevents features with larger values from dominating the model.

Click to watch →

Deep Dive

Regularization techniques, such as Lasso (L1) and Ridge (L2) regression, to prevent overfitting in trained models.

Machine Learning 02 | Linear Regression | DA | GATE Crash Course

Machine Learning 02 | Linear Regression | DA | GATE Crash Course

13.8K

Regularization reduces overfitting by adding penalty terms to the cost function. Lasso (L1): Cost = Σ(yi - yhat)² + λ×Σ|m|, can shrink coefficients to zero (feature selection). Ridge (L2): Cost = Σ(yi - yhat)² + λ×Σm², reduces coefficient magnitude but not to zero. λ controls regularization strength. Higher λ means more shrinkage. These techniques help create models that generalize well to new data by preventing over-specialization to training data.

Click to watch →

Lec-37: Ridge and Lasso Regression | Machine Learning

Lec-37: Ridge and Lasso Regression | Machine Learning

85.8K

Ridge Regression (L2 regularization) and Lasso Regression (L1 regularization) are techniques used in linear regression to prevent overfitting by adding penalty terms to the cost function; Ridge adds λ multiplied by the sum of squared coefficients (λ × Σβ²), while Lasso adds λ multiplied by the sum of absolute coefficients (λ × Σ|β|), with λ controlling the strength of the penalty between 0 and 1, where higher λ values impose stronger regularization to reduce model complexity and improve generalization on unseen data.

Click to watch →

Regularization In Machine Learning | Regularization Example | Machine Learning Tutorial |Simplilearn

Regularization In Machine Learning | Regularization Example | Machine Learning Tutorial |Simplilearn

29.9K

Regularization prevents overfitting and underfitting by adding penalty terms to loss functions. Ridge (L2) adds the sum of squared coefficients: Cost = Loss + λΣ(w²), shrinking coefficients toward zero without eliminating them. Lasso (L1) uses absolute values: Cost = Loss + λΣ(|w|), potentially setting coefficients exactly to zero for feature selection. Ridge suits high-dimensional data with many variables; Lasso works better with fewer variables and enables feature selection. The choice depends on dataset characteristics and modeling goals, balancing model complexity against generalization ability.

Click to watch →

HIMTI Responsi 2026/2027 | Machine Learning [B28] | UTS Semester Genap

HIMTI Responsi 2026/2027 | Machine Learning [B28] | UTS Semester Genap

533

Regularization is a technique to reduce overfitting by adding a penalty term to the model's loss function, which discourages complex models. Lasso (Least Absolute Shrinkage and Selection Operator) regression uses L1 penalty (absolute values of coefficients) and can set some coefficients to exactly zero, effectively performing feature selection. Ridge regression uses L2 penalty (squared values of coefficients) and shrinks coefficients toward zero but rarely sets them to exactly zero. Both techniques reduce model complexity and improve generalization.

Click to watch →

What is Regularization | Model Over-fitting | Lasso | Ridge Regression

What is Regularization | Model Over-fitting | Lasso | Ridge Regression

45K

Regularization is a technique in machine learning that prevents overfitting by adding a penalty term to the cost function, which constrains the model parameters to remain small and normal. The two main types are L1 regularization (Lasso) that adds the sum of absolute values of parameters, promoting feature selection by setting unimportant feature weights to zero, and L2 regularization (Ridge) that adds the sum of squared parameters, which shrinks all weights but rarely produces exact zeros. Both methods help improve model generalization on new data by reducing sensitivity to parameter changes.

Click to watch →

Cost Function & GD Intro

GD Step Mechanics

Slope Direction Logic

Data & Matrix Setup

Prediction Formula

Vectorized Cost Derivative

GD Pseudo-code

Learning Rate Effects

Key Takeaway

Cost Function & GD Intro

0:00

Playing Section

1
Linear regression aims to fit a best-fit line by minimizing error.
2
Cost function measures prediction error; smaller is better.
3
Gradient descent is the algorithm used to minimize this cost.

112.4K views1.8K17:59@MachineLearningWithJayOriginal Release: 2020-07-26

Gradient Descent is an optimization algorithm used in Linear Regression to minimize the cost function, which measures the error between predicted and actual values. The algorithm works by iteratively updating the parameters (theta) using the formula: theta = theta - alpha × (derivative of cost with respect to theta), where alpha is the learning rate. By repeatedly applying this update rule, the algorithm converges toward the minimum of the cost function, allowing the regression line to best fit the data set.

[00:00:00]what is going on everybody this is jp and welcome to code boosters in this video we are going to talk about the linear regression gradient descent now we know that the linear regression makes the prediction by plotting a straight line that fits best to the data set and the way to plot this straight line is by using something called cost function which is the measurement of how well the predictions are doing or in other words cost function or the cost value represents the error between the actual predictions and the actual value and in the previous video we saw that the cost function formula is given by this where m is the total number of data points y is the actual value or the labels and why the hat is are the predictions so in order for the straight line to fit the plot the cost function or the error representation should be minimum and to minimize this we need to use the gradient descent algorithm so the gradient descent algorithm is the algorithm we use to minimize the cost function value so that our straight line fits best to the data set so what is gradient descent algorithm let us start let us say we are plotting cost versus theta graph here i am taking only one of the many theta parameters if the number of theta parameters increases the dimension of the graph increases so for simplicity i'm only taking one of the theta parameter and i'm plotting it with respect to the cost the graph for the cost and the theta is going to look like this a parabola now let's say at the beginning we had a cost function value which was here so our goal is to minimize this cost function the minimum of the cost lies at its minimum so we need to reach from this point to its minima how do we do that let us say we take one step and we reach here at this point then we take another step and then we reach even closer at this point and then we we take another step we reach here but we are exceeded so we'll backtrack reach here then we'll reach here then we will oscillate around this region now one thing to know with the gradient descent algorithm we are taking one step we are taking steps like this to reach to its minimum as we are taking steps every time we are never going to be able to converge exactly at its minimum but we are going to oscillate the region very close to that minima but that serves our purpose now what we do in gradient descent is that we repeat a step which is theta equals to theta minus alpha times derivative of cost with respect to derivative of theta okay so we repeat this equation it is alpha multiplied by derivative of cost versus with respect to derivative of theta and alpha here is a positive constant now why does this step or this equation represents one step of this algorithm let us take a closer look let us examine this derivative of cos with respect to derivative of theta when we are at this point or this half of the parabola the derivative of cost with respect to derivative of theta which is nothing but the slope of the graph will be negative okay so slope will be negative right so this quantity is going to be negative multiplied by negative sign which is going to be positive so theta was here sorry so the theta was here but as theta is adding with some positive quantity negative negative multiplied by negative positive theta is adding with some positive quantity it is going to increase okay did you see okay and let us say if we are on this half right here then the slope is going to be positive so negative minus positive quantity so the theta is going to actually decrease so every time we take this equation every time we repeat this equation if it if the initially theta was on this side the theta is going to go closer and closer and closer towards the minima and if the theta becomes let's say exceeded to here like in this step which exceeded it is going to backtrack or it is it is going to backtrack and it is going to again go closer and closer to the minimum so when we repeat this step for quite a number of times we eventually end up to the local minima that was the gradient descent algorithm so now let us derive the derivative of the cost function with respect to theta before that let us see how our data set looks like let's say our data set is x and it has m data points and n dash features okay so the data set will be in the form of matrix of m comma n dash now i'm going to do something different with this data set and that is i'm going to add another column vector of ones to the data set right so the total dimension or the total number of columns or the total number of features are going to be n dash plus 1.

[00:06:57]let's say that is n now why did i do that it will be more clear in a while similarly the the labels y will have m data points one for each the spelling of data points is wrong but bear with it one for each label right so the dimension of y will be m comma one now we know that the predictions y hat has the formula theta n dash x n dash plus theta n dash minus 1 x n dash minus 1 plus dot theta 1 x 1 plus theta naught so if we were to get a vector for the theta it will be of the dimension n dash plus 1 comma 1 n plus 1 is nothing but n so the dimension or the size of the theta array or vector will be n comma 1 okay so let us let me summarize x is our data set where on the column side there are features and let us save the data set is of house prices the features can be size of the house number of bedrooms square foot area of the house or the number of other rooms bathroom quality living room quality etc or garage number of cars it can store etc on the roadside it is going to be the features for each house let's say we have data of m house for every house they're going there is going to be one prediction which is the price of the house which is stored in the y label and linear regression the predictions are given by this formula so theta and these thetas are parameters and x are the features of the house so if we if i write the vector of theta in this form it will be of size n comma one i hope this is very clear okay so why we are discussing this you see the equation of prediction is given by this and this represents only one prediction for one house where we have the values or the features of the house given by x's and thetas as the parameter so if we want to represent the predictions in the form of metrics the predictions care for m houses can be represented by nothing but x matrix multiplication theta how the size of the x is n comma n size of theta is n comma 1 which gives us this equation for m times so this overall size is going to be one so y hat will be again a matrix of these equations for different different houses for m houses okay now we know the prediction can be represented in the matrix form as x matrix multiplication theta now the cost function has the equation as this we can compute the value of y or we can compute y hat here as summation of the matrix of y this is capital y minus of this y hat capital y hat and square okay summation of this i hope you can understand that so what i did here is that i represented this summation in the matrix form so y is the matrix of the actual labels and y hat are the matrix of the predictions and we squared it and we are just summing the resultant matrix this is the cost okay now now the formula for the cost function is given by this so the del cost by del theta will be if you know the calculus then you will know the del cos by del theta will come e as 1 by m matrix multiplication of x transpose n y minus y hat now how did this came if you know calculus you will see that the derivative of this term will remove the square and it will cancel this two and then we will have a partial derivative of this term which will bring only x and as it is in the matrix multiplications the dimensions must match so the dimension of x transpose is n comma m and the dimension of the this is m comma 1 so the oral dimension will be n comma 1 which is the dimensions of the theta vector okay so del cos by del theta is given by this equation now we are going to completely implement the gradient descent algorithm at the beginning we have our data set x the labels y and we initialize theta vector as all zeros and this is as the size of n comma one we are initializing theta as all zeros and we have the data set x and their labels y so as we have initialized theta zeros it will be somewhere near a very large cost value let's say i'm taking it here okay then we are going to loop this is a pseudo code for the gradient descent algorithm okay so we are going to loop let's say thousand times it can be any number of times as long as we reach to the minimum okay let's say we lose let's say we loop through a thousand times we are going to first make find out the predictions which is given by matrix multiplication of x and sorry this is theta we're going to do the matrix multiplication of x and theta then we're going to find out the cost function value which is given by this sorry there's a square here then we are going to find the derivative of del derivative of cost with respect to theta i'm representing as del theta so del theta is the derivative of cost with respect to theta and we just saw it is it is equal to 1 by m matrix multiplication of x transpose and y minus of y hat and then we will update theta as theta minus of alpha which is a positive constant by a multiplied by del theta so del theta is going to be negative as the slope is negative so theta is going to increase so theta will go from here to somewhere here and we will move here similarly as the loop goes and goes we will eventually move closer and closer towards the minima and when it reaches here the slope will be positive so the theta will decrease and it will oscillate at a very close region near its minima and that's how we minimize the cost function so this was the complete implementation of the gradient descent algorithm now one thing to note is about alpha here here alpha is a positive constant and we call it learning rate it is called learning rate and there's a reason why it is called learning rate and the reason is the value of alpha determines how much bigger our step is going to be so if the value of alpha is large we're going to take larger steps but if the value of alpha is small we're going to take smaller smaller steps now if the value of alpha is extremely large then it is going to overshoot and what do i mean by that i mean from here if we were to converge and the alpha is very high instead of going this side it will go directly here up and from here it will go up so basically if the value of alpha is extremely large it is instead of getting converging it is going to diverge and it will go up to infinity so that was something to be noted about alpha okay that is so good you just saw the gradient descent algorithm and you also saw the implementation or the pseudo code of the gradient descent algorithm now if you want to sustain it you must implement the complete linear regression model by yourself otherwise it's just going to up the mind so i have another video for you which shows the complete implementation of the linear regression with python and the way we are going to implement this by making a model on house price predictions you're going to love that now if you're interested in learning machine learning then hit the red subscribe button because i upload a machine learning video every week and if you are kind of a student like me or you like a fun learning then you might like my channel so hit the red subscribe button also if you like the video give this a thumbs up hit the bell icon as well and i see you in the next one

Up Next

Backpropagation & Neural Networks | Stanford CS224N Lecture 3

@stanfordonline

117.8K views•2021-10-29

Building Real-Time ML Pipelines with Feature Stores and MLOps Frameworks

@ODSCAI

5.1K views•2022-02-20

Bypassing Tor Censorship: Bridges and Pluggable Transport Guide

@Coding_ForEveryone

397 views•2024-06-11

Neural Networks Explained: Math, Layers, and Learning Fundamentals

@3blue1brown

21.9M views•2017-10-05

Related Study Plans & Knowledge Roadmaps

Structured learning paths in Artificial Intelligence

View All Artificial Intelligence Topics

artificial-intelligence

Building Recommenders: Two-Tower & Vector DBs

01Foundations of Python & Machine Learning Basics

02Introduction to Recommender Systems

03Deep Learning & Representation Learning

+ 3 more learning modules

Explore Roadmap

artificial-intelligence

Latent Diffusion: PyTorch, UNet & CLIP

01Deep Learning & PyTorch Foundations

02Convolutional Networks & Autoencoders

03The Math and Mechanics of Diffusion Models

+ 3 more learning modules

Explore Roadmap

artificial-intelligence

NeRFs: 3D Reconstruction & Instant-NGP

01Math & Python Foundations for 3D Vision

02Photogrammetry & Camera Pose Estimation

03Neural Radiance Fields (NeRF) Core Concepts

+ 2 more learning modules

Explore Roadmap

artificial-intelligence

TFT Forecasting: Energy Demand & Deep Learning

01Programming & Data Science Foundations

02Introduction to Machine Learning & Time Series

03Deep Learning & Sequential Models

+ 3 more learning modules

Explore Roadmap