Spectral Clustering: Three Steps to Graph Partitioning | Stanford Lecture 34

Linear algebra studies vector spaces—sets with addition and scalar multiplication satisfying specific axioms. Examples include R², R³, Rⁿ, complex numbers C, and polynomials of degree 1. A key insight is that polynomials are vectors because they satisfy vector space properties, demonstrating vectors are more general than geometric arrows. Linear transformations preserve addition and scalar multiplication between spaces. Spaces with the same dimension are isomorphic, meaning they are algebraically identical. The derivative operation on polynomials (Ax + B → A) is a linear transformation representable as a matrix [[0, 1]]. Linear functionals map vector spaces to their associated fields. In finite-dimensional spaces, every linear transformation corresponds to a matrix. Linear algebra is essential for Calculus 2, Calculus 3, and Differential Equations. Key concepts include recognizing vector spaces, linear functionals, dual and bidual spaces, and inner products. Recommended resources include books by Elon Lages, Flávio Ulhoa, Gilbert Strang, and Hoffman.

In linear algebra, a set of vectors is linearly independent if no non-trivial linear combination of them equals the zero vector; a basis for a vector space is a set of vectors that is both linearly independent and spans the entire space, and the dimension of a vector space is defined as the number of vectors in any basis for that space, which remains constant regardless of which basis is chosen.

Linear algebra is a fundamental module in engineering mathematics, particularly important for GATE examinations. It consists of 2-3 marks questions annually. The subject is also known as 'matrices' and involves solving systems of simultaneous equations. Matrices have wide applications in technical subjects including fluid mechanics (stress tensors), finite element analysis (stiffness matrices), and various engineering disciplines. The instructor recommends learning in English as it is the primary language of education worldwide, even at top institutions like IITs and IISc Bangalore.

Linear algebra studies systems of linear equations, which are equations of the form A₁X₁ + A₂X₂ + ... + AnXn = B, where none of the variables are raised to any exponent or involved in products/quotients. These systems model real-world relationships because they strike a balance between simplicity and complexity—simple enough to be solvable yet complex enough to capture many practical scenarios. A solution to a system is a set of values for each variable that satisfies all equations simultaneously. When graphed, the solution corresponds to the intersection point of lines (for two variables) or planes (for three variables). Systems can have a unique solution, no solution (inconsistent), or infinitely many solutions (dependent), depending on whether the equations intersect at a point, are parallel, or describe the same line/plane.

A matrix is a rectangular table of numbers organized in rows and columns, classified by dimensions (m×n) and special types including row matrices, column matrices, unitary matrices, rectangular matrices, square matrices (where m=n, called 'ordem'), triangular matrices (lower/upper), diagonal matrices, and identity matrices (where all diagonal elements equal 1); matrices can be defined through formation laws using functions of their indices, and basic operations like equality, addition, and subtraction require matrices to have identical dimensions, with addition/subtraction performed element-wise.

Five primary clustering algorithms exist: (1) K-Means divides data into K groups using random centroids and iterative refinement, (2) Mean Shift automatically determines cluster centers by moving points toward density peaks, (3) DBSCAN identifies clusters based on density and automatically detects noise/outliers, (4) OPTICS extends DBSCAN for varying densities using core and reachability distances, and (5) Hierarchical clustering builds dendrograms showing nested relationships. Each algorithm has strengths for different data types and use cases.

There are two major categories of clustering algorithms: (1) Hierarchical clustering, which creates tree-like structures and includes agglomerative (bottom-up merging of elements into larger clusters) and divisive (top-down splitting of the whole set into smaller clusters) approaches, and (2) Partitional clustering, which divides data into non-overlapping subsets. K-Means is a partitional clustering method where objects are divided into exactly K clusters based on their characteristics. Fuzzy C-Means is similar but allows objects to belong to multiple clusters simultaneously, unlike K-Means where each object belongs to only one cluster.

Clustering is an unsupervised machine learning technique that groups unlabeled data into natural clusters by minimizing dissimilarity, defined as the sum of variabilities within each cluster. Two primary algorithms are presented: hierarchical clustering, which starts with each item as its own cluster and iteratively merges the most similar pairs using linkage metrics (single-linkage, complete-linkage, or average-linkage), producing a dendogram for visualization; and k-means clustering, a faster greedy algorithm that randomly initializes k centroids, assigns examples to nearest centroids, recomputes centroids as averages, and repeats until convergence. Key considerations include choosing appropriate constraints (such as limiting cluster count), handling feature scaling to prevent range-dominated features from skewing results, addressing sensitivity to initial conditions through multiple trials, and determining optimal k values through experimentation or hierarchical analysis on subsets.

Clustering algorithms group similar data points together without labeled outcomes. K-Means iteratively assigns points to centroids and updates centroids as means, converging to local optima. Hierarchical clustering builds trees of clusters by repeatedly merging closest clusters (agglomerative) or splitting largest clusters (divisive), useful when the number of clusters is unknown. DBSCAN identifies clusters based on density, finding regions of high density separated by low-density regions, capable of discovering arbitrary-shaped clusters and identifying outliers. Each algorithm has strengths depending on the data characteristics and application requirements.

Beyond k-means, two important alternatives are: (1) DBSCAN - effective for non-convex, anisotropic clusters; (2) Agglomerative Clustering - hierarchical approach that builds clusters from the bottom up. These are preferred when data doesn't form spherical clusters, though k-means is faster and more efficient for normally distributed data.

The graph Laplacian is defined as L = D - A, where D is the degree matrix (diagonal with node degrees) and A is the adjacency matrix. When applied to node features, it computes the sum of differences between a node and its neighbors: Lx_i = x_i - Σ(x_j for j connected to i). This discrete operator mirrors the continuous Laplacian in physics, which measures how different a point is from its surroundings, enabling applications in heat diffusion and graph analysis.

The graph Laplacian measures deviation from local average: for a graph with vertex values u_i, Lu_i = (average of neighboring values) - u_i. This represents how much a node differs from its friends' average. The continuous Laplacian generalizes this: Δu(x₀) = lim(ε→0) [∫_{B(x₀,ε)} u(x) dA / Area(B(x₀,ε)) - u(x₀)]. As ε→0, this deviation becomes exactly the Laplacian, providing intuitive understanding of the operator as measuring local averaging behavior.

Spectral graph partitioning uses the graph Laplacian matrix to identify clusters or communities in networks by finding eigenvectors that reveal natural divisions in the graph structure; the key insight is that multiplying the adjacency matrix by a vector sums the values of neighboring nodes, and the eigenvectors of this matrix correspond to meaningful partitions where nodes within clusters have similar values and nodes across clusters have different values.

This section introduces the graph Laplacian as a fundamental operator for spectral graph theory. The instructor defines the graph gradient as ∇f(i) = Σⱼ wᵢⱼ (f(j) - f(i)), measuring how a function changes in the neighborhood of vertex i. The graph divergence is defined as div(f)(i) = Σⱼ wᵢⱼ (f(j) - f(i)), representing net flow into or out of a vertex. The graph Laplacian is then defined as Lf(i) = -div(∇f)(i) = Σⱼ wᵢⱼ (f(i) - f(j)), measuring how much a function value differs from the weighted average of its neighbors. The section establishes that the Laplacian can be represented as a matrix L = D - W, where D is the degree matrix and W is the adjacency matrix.

The graph Laplacian L = D - W combines the degree matrix D and weight matrix W of a weighted undirected graph. Its diagonal entries are vertex degrees (positive), and off-diagonal entries are negative weights or zero. Key properties include: row sums equal zero, eigenvalues are real and non-negative, and the smallest eigenvalue is always zero with eigenvector being the vector of all ones. The sum of squares theorem shows X^T L X equals half the sum of squared differences across edges, proving the Laplacian is positive semi-definite.