PyTorch Deep Learning Framework: Build Neural Networks from Scratch

Vectors are 1D arrays (np.array([1, 2, 3])). Matrices are 2D arrays (np.array([[1, 2, 3], [4, 5, 6]])). Tensors are multi-dimensional arrays with more than two dimensions (np.array([[[1, 2], [3, 4]], [[5, 6], [7, 8]])). The instructor explains that tensors are used in deep learning frameworks like TensorFlow because they can represent complex data structures efficiently.

In linear algebra, mathematical objects are categorized by their dimensionality: scalars are single numbers (unbolded lowercase letters), vectors are ordered lists of numbers representing points in n-dimensional space (bolded lowercase letters), matrices are two-dimensional grids of numbers (bolded uppercase letters), and tensors are multi-dimensional arrays extending beyond matrices (bolded uppercase letters). Each object type has specific notation conventions and set declarations (e.g., ℝⁿ for n-dimensional vectors, ℝᵐ×ⁿ for m×n matrices).

Linear algebra provides mathematical frameworks for representing and manipulating data essential for artificial intelligence. Scalars are single numerical values on a number line. Vectors are ordered collections of scalars arranged in one dimension, representing quantities with both magnitude and direction. Matrices are two-dimensional arrays of vectors, forming rectangular grids. Tensors generalize these concepts to higher dimensions. All real-world phenomena can be represented as mathematical data: images as pixel grids with RGB values, text as character sequences, and numerical values as scalars. The fundamental operations in linear algebra are addition/subtraction and scaling (multiplication by scalars). Linear independence determines whether vectors can span their entire space—two vectors in 2D are independent if they point in different directions. Basis vectors are linearly independent vectors that span an entire space—exactly n vectors are needed for n-dimensional space. Matrix rank measures the maximum number of linearly independent rows or columns in a matrix. A matrix with full rank spans its entire row or column space. Rank reduction occurs when vectors are linearly dependent. Matrix-vector multiplication follows dimensional rules: an m×n matrix multiplied by an n×1 vector produces an m×1 vector. The null space consists of all vectors x satisfying Ax = 0, revealing when matrices collapse dimensions.

This lesson introduces four fundamental mathematical structures used in linear algebra and computer science. A scalar is a single number from any set (natural, integer, rational, or irrational). A vector is a one-dimensional collection of scalars, represented by lowercase letters, with dimensions and positions. A matrix is a two-dimensional array of scalars, represented by uppercase letters, with rows and columns indexed from zero. A tensor is the logical extension of these structures: zero-dimensional tensors are scalars, one-dimensional are vectors, two-dimensional are matrices, and three or more dimensions are also called tensors. These structures form the foundation for understanding more complex mathematical operations in data science and machine learning.

Vectors are unidimensional arrays holding one-to-n elements in a single direction. Matrices organize elements in rows and columns, distributing data across two dimensions. Tensors generalize this concept to two or more dimensions, enabling representation of artificial neurons with hundreds of thousands of dimensions. Contemporary AI models may employ millions of dimensions, requiring purely mathematical representation since such quantities cannot be visualized directly.

Three essential calculus concepts underpin backpropagation: (1) Derivative measures the slope of a tangent line to a curve at a specific point, representing the rate of change of a function—for example, the derivative of x² is 2x; (2) Partial derivative extends this concept to functions of multiple variables by measuring change with respect to one variable while holding others constant; (3) Chain rule enables computation of derivatives for composite functions formed by nesting multiple functions together.

This comprehensive section covers the mathematical foundations essential for understanding gradient descent and neural network optimization. Derivatives measure instantaneous rates of change—the slope of a tangent line at a specific point, formally defined as f'(a) = lim(h→0) [f(a+h)-f(a)]/h. For linear functions, derivatives are constant; for nonlinear functions like x², derivatives vary with x (power rule: d/dx [x^n] = n×x^(n-1)). Partial derivatives extend differentiation to multivariable functions, measuring how functions change when one variable changes while others remain constant. The gradient vector combines all partial derivatives into a direction vector pointing toward fastest increase. The chain rule handles composite functions: if y = f(g(x)), then dy/dx = (dy/dg) × (dg/dx). These concepts form the mathematical machinery needed to compute gradients for optimizing complex machine learning models.

The chain rule enables differentiation of composite functions by multiplying derivatives along dependency chains, forming the mathematical foundation for backpropagation in neural networks. Directional derivatives measure rate of change in any arbitrary direction specified by a unit vector. The gradient vector contains all first-order partial derivatives and points in the direction of maximum increase. The directional derivative equals the dot product of the gradient and direction vector. Gradient descent moves in the negative gradient direction to find minimum points, as the maximum decrease occurs opposite to the gradient direction. This principle underlies optimization in machine learning.

Backpropagation reverses the forward pass to propagate error backward through the network. Partial derivatives are calculated to understand how weight changes affect error. For y = x² + c, the derivative is 2x. The chain rule enables computing gradients: ∂Error/∂Weight = (∂Error/∂Output) × (∂Output/∂Weight). For squared error: ∂Error/∂Output = (Target - Output). For output calculation: ∂Output/∂Weight = Input. Combining these gives ∂Error/∂Weight = (Target - Output) × Input. For sigmoid activation: ∂Output/∂Input = Output × (1 - Output). These derivatives enable precise weight adjustments throughout the network.

Differential calculus studies how functions change with respect to parameters, with the derivative representing the instantaneous rate of change. Key concepts include basic differentiation rules (constant, power, sum, product, quotient), partial derivatives for multivariable functions, and the chain rule for composite functions. In machine learning, these concepts are applied through gradient descent, where the gradient (vector of partial derivatives) indicates the direction of steepest ascent, and parameters are updated in the opposite direction to minimize loss functions. The chain rule enables efficient gradient computation in neural networks by propagating derivatives through multiple layers, forming the mathematical foundation for backpropagation algorithms.

Machine learning begins with data as fuel for models. Data consists of features (input columns for prediction) and labels (output columns to predict). In supervised learning, both features and labels are provided during training so the model learns relationships. After training, only features generate predictions. The workflow includes: defining problems, collecting/preparing data (converting text to numerical, cleaning, handling missing values, encoding categories, splitting into train/test sets), selecting models, training, evaluating accuracy, tuning, final testing, and deployment. Train-test split is critical: 70-80% diverse data for training prevents overfitting, while 20% unseen data tests accuracy. Supervised learning categorizes into classification (categorical outputs like spam/not spam) and regression (numerical outputs like house prices).

A complete supervised learning training loop consists of several essential components: (1) sampling a batch of data from the training set, (2) computing model predictions (log probabilities), (3) calculating the loss using a loss function like negative log likelihood, (4) calling zero_grad() to reset gradients, (5) performing backpropagation via loss.backward(), and (6) executing optimizer.step() to update model weights.

Supervised machine learning involves learning a function that maps input features (x) to output labels (y) from labeled training data, where the true function is unknown. The workflow includes: data preprocessing (feature extraction, normalization, splitting into training/test sets), model training using algorithms like linear regression, decision trees, or neural networks, and model evaluation using metrics such as accuracy, precision-recall, or loss functions. Data is represented as a design matrix where rows are samples and columns are features. Algorithms can be classified as parametric (fixed number of parameters) or non-parametric (parameters grow with data), and as eager (create model during training) or lazy (compute during prediction).

Supervised learning is the most common and fundamental type of machine learning. In supervised learning, the computer is trained by providing it with examples along with their correct answers. The computer learns to map inputs to outputs by studying these labeled examples. 'Features' are the characteristics or attributes that describe an object (like number of rooms, area, location for a house). 'Labels' are the desired outputs or answers that the model should predict (like house price). In supervised learning, both features and labels are provided during training so the computer can learn the relationship between them. After learning, when only features are provided, the computer can predict the corresponding label. This allows the model to generalize from training data to make predictions on new, unseen data.

Machine Learning is a system that learns to combine input data to make accurate predictions about unseen data. In supervised learning, the system uses labeled data where each input (features) has a corresponding output (label) to learn from. Features are the input variables represented as vectors, while labels are the target values the system attempts to predict. The system learns by comparing its predictions with actual labels and adjusting its approach accordingly. Mathematical notation uses lowercase letters for scalars, bold or underlined letters for vectors, and uppercase letters with double lines for matrices, with dimensionality representing the number of features in the input vector.