Machine Learning: Algorithms & Optimization

Learning Goal: Master the conceptual and practical implementation of supervised learning, gradient descent optimization, training techniques, and comprehensive model evaluation and predictive modeling.

• Estimated Total Study Time: 35 Hours
• Prerequisites: Basic familiarity with Python and high school-level algebra. No prior machine learning experience is required.

---

Module 1: Introduction to Machine Learning & Core Math

This module establishes the conceptual foundation of machine learning, clarifying the fundamental distinctions between supervised and unsupervised paradigms. It also builds the necessary mathematical framework in linear algebra, vector spaces, and matrix operations to ensure you can follow the algorithmic mechanics in later modules.

Recommended Videos

• Why this video: This video directly bridges the conceptual gap highlighted in our curriculum review. It provides a highly visual, mid-length conceptual overview of supervised and unsupervised learning techniques (such as regression and basic decision trees), giving you an intuitive mental model before diving deep into mathematical formulas.

Knowledge Checkpoint

• Understand the core difference between supervised learning (using labeled data for targets) and unsupervised learning (discovering underlying patterns or clusters in unlabeled data).
• Identify key applications of regression versus classification.

---

• Why this video: A comprehensive, classroom-style breakdown of the exact mathematical machinery needed for ML. It covers vectors, matrices, dot products, basic derivatives for optimization, and statistical fundamentals.

Knowledge Checkpoint

• Perform basic matrix addition, subtraction, and multiplication.
• Explain how a dot product represents similarity or projection between two vectors.
• Understand how a derivative represents the rate of change of a cost function.

---

• Why this video: This video demonstrates why matrix operations are so powerful in machine learning. It walks through representing neural networks and system equations as single-line matrix calculations, allowing efficient parallel computation of predictions across thousands of data points.

Knowledge Checkpoint

• Express a system of linear equations or weights as a matrix-vector multiplication.
• Understand how weight matrices map input dimensions to hidden layer dimensions.

---

Module 2: Supervised Learning: Regression Models

This module focuses on modeling continuous and categorical predictions. You will master Simple and Multiple Linear Regression, R-Squared evaluation, and transition to Logistic Regression for binary classification.

Recommended Videos

• Why this video: A legendary visual explanation of Simple Linear Regression. Josh Starmer details how we calculate the line of best fit using the Least Squares method and conceptually evaluates residuals.

Knowledge Checkpoint

• Define a "residual" and explain how the Least Squares method minimizes the sum of squared residuals.
• Interpret the slope and intercept coefficients of a linear model.

---

• Why this video: A quick and highly conceptual extension of linear regression into multiple dimensions. This explains how models handle several input features simultaneously by fitting planes and hyperplanes instead of simple 2D lines.

Knowledge Checkpoint

• Distinguish simple linear regression from multiple regression.
• Understand how adding predictor variables changes the geometric shape of the decision boundary.

---

• Why this video: Transition from continuous prediction to classification. This video explains how logistic regression maps values from $-\infty$ to $+\infty$ onto a probability scale of $0$ to $1$ using an S-shaped sigmoid curves and utilizes Maximum Likelihood for parameter optimization.

Knowledge Checkpoint

• Explain why standard linear regression is unsuitable for binary classification tasks.
• State the formula and output range of the Sigmoid activation function.
• Understand the basic concept of Maximum Likelihood as a cost function for logistic regression.

---

Module 3: Gradient Descent and Model Optimization

Optimization is the heart of machine learning. In this module, you will learn how models "learn" parameters using partial derivatives, cost functions (Mean Squared Error), and gradient descent variants.

Recommended Videos

• Why this video: Unmatched visualization of how high-dimensional error landscapes are traversed. It treats gradient descent as rolling down a loss landscape to find minimum error points, helping you build an absolute mental model of optimization.

Knowledge Checkpoint

• Define "gradient" and explain why we move in the opposite direction of the gradient vector.
• Identify the hazard of setting a learning rate too high (divergence) vs too low (slow convergence).

---

• Why this video: Concrete code implementation. This guide builds a gradient descent algorithm in Python from scratch, showing step-by-step calculations of the cost function (MSE) and parameter updates.

Knowledge Checkpoint

• Write down the objective function for Mean Squared Error (MSE).
• Implement parameter updates ($\theta = \theta - \alpha \cdot \text{gradient}$) in a basic program structure.

---

• Why this video: This video breaks down the calculus behind the curtain, walking through the power rule and chain rule calculations of the partial derivatives for $m$ (slope) and $b$ (y-intercept).

Knowledge Checkpoint

• Calculate the partial derivatives of the MSE cost function with respect to weights and bias.
• Differentiate between Batch Gradient Descent, Stochastic Gradient Descent (SGD), and Mini-batch Gradient Descent in terms of updates per epoch.

---

Module 4: Advanced Supervised Learning Algorithms

Moving beyond simple linear splits, this module explores non-linear, robust classification and regression algorithms, featuring decision-making trees, ensembles, and high-margin classifiers.

Recommended Videos

• Why this video: Learn the fundamental logic of decision tree splits. This video explains Gini Impurity, entropy, and step-by-step leaf construction for handling both continuous and categorical variables.

Knowledge Checkpoint

• Explain how Gini Impurity is used to choose features and split points in a decision tree.
• Differentiate between a root node, internal decision node, and terminal leaf node.

---

• Why this video: Introduces the concept of Ensembles. This video covers bootstrapping data, feature bagging (subsetting features), and aggregating trees into a highly generalized Random Forest.

Knowledge Checkpoint

• Explain "bagging" (bootstrap aggregating) and how it reduces the variance of individual decision trees.
• What is Out-Of-Bag (OOB) error and how is it calculated?

---

• Why this video: Address the critical gap identified in our review: a practical, complete implementation of Support Vector Machines (SVM). This comprehensive guide works through imports, handling missing data, parameter tuning ($C$, $\gamma$), and applying the Radial Basis Function (RBF) kernel in scikit-learn.

Knowledge Checkpoint

• Explain the concept of a "hyperplane" and "margins" in SVM.
• Define the "Kernel Trick" and how it projects non-linearly separable data into higher dimensions.
• Identify how parameter $C$ controls the tradeoff between margin size and misclassification penalties.

---

• Why this video: Understand simple instance-based lazy learning. This walks through KNN's core mechanism—calculating Euclidean distances between a query point and existing data to make a majority-vote classification.

Knowledge Checkpoint

• Explain why KNN is considered a "lazy learner."
• Compute Euclidean distance between two points in 2D space.
• Discuss how choice of $k$ affects model overfitting (small $k$) vs underfitting (large $k$).

---

Module 5: Model Evaluation & Generalization

Building models is only half the battle; ensuring they generalize to unseen real-world data is crucial. This module covers bias-variance trade-offs, regularization methods (Ridge/Lasso), cross-validation, and classification performance metrics like confusion matrices, precision, recall, and F1-score.

Recommended Videos

• Why this video: A rigorous academic deep-dive from Stanford University. It presents the formal statistical formulations of bias and variance and outlines the mathematics of why adding L1/L2 penalties prevents overfitting.

Knowledge Checkpoint

• Define "high bias" (underfitting) and "high variance" (overfitting).
• Explain how regularization penalizes model complexity to lower variance.

---

• Why this video: This video covers fundamental evaluation metrics such as the Confusion Matrix, absolute error, sum of squares error, and misclassification rate, directly filling a core gap identified in our review.

Knowledge Checkpoint

• Construct and label a 2x2 Confusion Matrix (TP, FP, TN, FN).
• Explain the concept of $K$-fold cross-validation and why it protects against validation bias.

---

• Why this video: A solid, practical tutorial on using scikit-learn in Python to calculate evaluation metrics. It explicitly covers Precision, Recall, and the harmonic mean (F1 Score) in both binary and multiclass setups.

Knowledge Checkpoint

• Calculate Precision: $\frac{TP}{TP + FP}$.
• Calculate Recall: $\frac{TP}{TP + FN}$.
• Explain when F1-score is preferred over raw accuracy (e.g., highly imbalanced datasets).

---

• Why this video: Visualizes the shrinking of parameters in Ridge Regression. It explicitly explains the penalty term $\lambda \times \text{slope}^2$ and how it handles datasets with fewer observations than features.

Knowledge Checkpoint

• Explain the Ridge (L2) penalty term and how it prevents asymptotic explosion of regression coefficients.
• Differentiate between Ridge (L2) and Lasso (L1) regularization in terms of sparsity (setting coefficients to exactly zero).

---

Course Map

---

Key People Index

• Josh Starmer (StatQuest): Renowned educator who demystifies complex statistics, linear algebra, and ensemble trees using clear, intuitive visuals and minimal math jargon.
• Grant Sanderson (3Blue1Brown): Famous mathematical animator known for explaining deep geometric concepts, vector transformations, and the visual topology of gradient descent optimization.
• Andrew Ng (Stanford Online/DeepLearning.AI): Leading pioneer in ML education whose CS229 lectures provide the definitive mathematical framing of the bias-variance tradeoff and parameter regularizations.

---

Final Self-Assessment

Test your mastery of the complete curriculum by ensuring you can perform or explain the following tasks:

• Explain the practical operational difference between supervised and unsupervised models.
• Calculate simple vector additions and matrix-vector products by hand.
• Formulate a linear regression prediction equation and explain how residuals are calculated.
• Apply the Sigmoid function to map a model output to a classification probability.
• Mathematically explain why gradients are calculated using partial derivatives of a cost function.
• Contrast Stochastic Gradient Descent (SGD) and Batch Gradient Descent in terms of memory load and optimization path stability.
• Understand how a Gini impurity reduction determines feature splits inside a decision tree.
• Explain the SVM Kernel Trick and how it manages high-dimensional classifications.
• Diagram a standard 2x2 confusion matrix and label TP, FP, TN, FN.
• Calculate Precision, Recall, and F1-score from a raw classification output.
• Explain the bias-variance trade-off and outline how L1 (Lasso) and L2 (Ridge) penalties adjust model weights to combat overfitting.