Spectral Clustering Explained: Theory & Key Steps

This comprehensive section covers the foundational concepts of graph theory. A graph consists of vertices (nodes/points) and edges (lines connecting vertices). Adjacent vertices are connected by an edge, while isolated vertices have no connections. Graphs can be undirected (no direction) or directed (edges have direction). Simple graphs have exactly one edge between any two vertices, while multigraphs allow multiple edges. Loops connect a vertex to itself. Weighted graphs assign costs to edges. The degree of a vertex is the number of edges incident to it, with self-loops counting twice. An isolated vertex has degree 0, while a pendant vertex has degree 1. The Handshaking Theorem states that the sum of all vertex degrees equals twice the number of edges, as each edge contributes to exactly two vertices. This theorem also implies that the number of odd-degree vertices must be even. Special graph types include complete graphs (n(n-1)/2 edges), regular graphs (all vertices same degree), and bipartite graphs (n²/4 maximum edges). Matrix representations enable algorithmic analysis: adjacency matrices use rows and columns for vertices with entries indicating edge presence, while incidence matrices use rows for vertices and columns for edges. For directed graphs, adjacency matrices capture edge direction, and incidence matrices use +1 for outgoing edges and -1 for incoming edges.

A graph G = (V, E) consists of vertices (nodes) and edges connecting them. Key concepts include: self-loop (edge connecting a vertex to itself), parallel edges (multiple edges between same vertices), simple graph (no self-loops or parallel edges), multigraph (allows multiple edges and self-loops), finite/infinite graphs (based on vertex/edge count), incident (edge connected to vertex), adjacent (vertices connected by edge), degree (number of incident edges), isolated vertex (degree zero), pendant vertex (degree one), and the Handshaking Theorem (sum of degrees equals twice the number of edges). Directed graphs (digraphs) have ordered pairs indicating direction, with in-degree and out-degree. Matrix representations are essential for computer applications: the incidence matrix (n×m) has 1 if edge j is incident on vertex i, 0 otherwise; the adjacency matrix (n×n) has 1 if edge exists between vertices i and j, 0 otherwise. For directed graphs, incidence matrices use +1 for outgoing edges and -1 for incoming edges.

Graph Theory is a mathematical concept for connecting and representing data elements. The basic components are vertices (nodes) and edges. A graph is formally defined as G = (V, E), where V is the set of vertices and E is the set of edges. Key concepts include: self-loop (edge from a vertex to itself), parallel edges (multiple edges between same vertices), trivial graph (one vertex, no edges), null graph (edges but no vertices), directed graph (edges with direction), simple graph (no self-loops or parallel edges), and multi-graph (contains parallel edges). Incidence refers to the relationship between an edge and the vertices it connects, while adjacency describes the relationship between two vertices connected by an edge. The degree of a vertex is the number of edges connected to it. In directed graphs, degree is divided into in-degree and out-degree. Special vertex types include isolated vertices (degree zero) and pendant vertices (degree one). Advanced classifications include weighted graphs (edges with numerical values), regular graphs (all vertices have same degree), and complete graphs (every vertex connected to every other vertex). Matrix representations provide computational tools: the adjacency matrix shows vertex connectivity, and the incidence matrix shows vertex-edge relationships. For directed graphs, the adjacency matrix is not symmetric.

A graph is a non-linear data structure consisting of vertices (nodes) and edges (connections). Graphs are classified into two types: (1) Undirected graphs where edges have no direction (bidirectional connections), and (2) Directed graphs where edges have specific direction (one-way connections). Adjacency matrix is a 2D array where rows and columns represent vertices. If an edge exists between vertex i and j, the cell (i,j) contains a value (1 for undirected, weight for weighted graphs). For directed graphs, the matrix is asymmetric. This representation allows O(1) edge existence checking but requires O(V²) space regardless of actual edges.

A graph consists of vertices and edges, with reachability asking whether a path exists between two vertices. Visual inspection is insufficient for computation; graphs must be represented mathematically. The adjacency matrix provides a systematic n×n representation where entry (i,j) equals 1 if there is an edge from vertex i to j, and 0 otherwise. For directed graphs, edges have direction; for undirected graphs, the matrix is symmetric. Neighbors are found by examining corresponding rows/columns. The degree of a vertex is the number of incident edges, ranging from 0 to n-1. In directed graphs, degree splits into in-degree and out-degree. This representation enables algorithmic manipulation of graph structures.

This comprehensive lecture covers matrix diagonalization, eigenvalues, and eigenvectors. Matrix diagonalization transforms a matrix into a simpler diagonal form where non-zero elements appear only on the main diagonal. The process involves finding eigenvalues and eigenvectors, constructing the model matrix P by placing eigenvectors as columns, and applying the formula AP P⁻¹. Eigenvalues are scalar values λ satisfying Av = λv, while eigenvectors are non-zero vectors whose direction remains unchanged. For an n×n matrix, there are n eigenvalues. Problems arise with repeated eigenvalues. Symmetric matrices (A = Aᵀ) have real eigenvalues and are always diagonalizable. Orthogonal matrices satisfy Aᵀ = A⁻¹. Eigenvalues have important properties regarding matrix powers: if A has eigenvalues λ₁, λ₂, ..., λₙ, then Aᵏ has eigenvalues λ₁ᵏ, λ₂ᵏ, ..., λₙᵏ. These concepts are fundamental to linear algebra and frequently tested in examinations.

Eigenvalues (λ) and eigenvectors (x) satisfy Ax = λx. For an n×n matrix A with eigenvalues λ₁ through λₙ and eigenvectors x₁ through xₙ, the relationship is AX = XD, where X contains eigenvectors as columns and D is diagonal with eigenvalues. The spectral decomposition states A = XDX⁻¹ when eigenvectors are linearly independent. A matrix is diagonalizable if and only if for each eigenvalue, its algebraic multiplicity equals its geometric multiplicity. The algebraic multiplicity is the number of times an eigenvalue appears as a root of the characteristic polynomial, while the geometric multiplicity is the dimension of the eigenspace.

This comprehensive section covers the foundational concepts of linear algebra: eigenvalues λ satisfy Av = λv for non-zero eigenvectors v, found by solving det(A - λI) = 0; eigenvectors solve the homogeneous system (A - λI)v = 0. A matrix is diagonalizable if it equals PDP⁻¹ with D diagonal, equivalent to having a basis of eigenvectors. Similar matrices share eigenvalues, while equivalence applies broadly. Orthogonal matrices satisfy QᵀQ = I with orthonormal columns. Symmetric matrices are always orthogonally diagonalizable (spectral theorem). These concepts enable spectral decomposition and form the mathematical framework for understanding matrix structure and transformation properties.

Matrix diagonalization is the process of transforming a square matrix A into a diagonal matrix D through similarity transformation, expressed as A = PDP⁻¹, where P is the modal matrix containing eigenvectors as columns and D is the spectral matrix containing eigenvalues on its diagonal. The process involves: (1) finding eigenvalues by solving the characteristic equation det(A - λI) = 0, (2) finding eigenvectors by solving (A - λI)x = 0 for each eigenvalue, (3) constructing the modal matrix P from eigenvectors, and (4) computing P⁻¹ to complete the diagonalization. A matrix is diagonalizable if it has n distinct eigenvalues or if the geometric multiplicity equals the algebraic multiplicity for each eigenvalue.

Diagonalization is the process of finding a basis in which a linear transformation has a diagonal matrix representation. An eigenvalue λ is a scalar such that there exists a non-zero vector v satisfying Av = λv. The equation (A - λI)v = 0 must have non-trivial solutions, which occurs when det(A - λI) = 0. The set of all eigenvalues is called the spectrum of A. To find eigenvalues, solve the system (A - λI)x = 0 for non-zero solutions. For matrix A = [[1,1,0],[1,1,0],[0,0,1]], the eigenvalues are λ = 0, λ = -1, and λ = 2. The corresponding eigenvectors are (1,-1,0), (1,1,-1), and (1,1,2). A matrix A is diagonalizable if there exists an invertible matrix P such that P⁻¹AP = D, where D is diagonal. The columns of P are the eigenvectors of A, and the diagonal entries of D are the corresponding eigenvalues. If A = PDP⁻¹, then Aⁿ = PDⁿP⁻¹, which simplifies computing high powers of matrices.

The RBF kernel measures similarity through exponential distance decay: K(X₁, X₂) = exp(-‖X₁ - X₂‖²/(2σ²)). Maximum similarity of 1 occurs when points coincide (distance=0), while increasing distance exponentially reduces similarity toward zero. Distance serves as an equivalent dissimilarity measure. Visual examples demonstrate how σ controls similarity region width: small σ creates narrow regions requiring extreme proximity, while large σ creates broad regions accepting distant points as similar.

The Radial Basis Function (RBF) kernel, also known as the squared exponential kernel, Gaussian kernel, or exponentiated quadratic kernel, is defined as k(x, z) = exp(-||x-z||²/(2σ²)), where ||x-z||² is the squared Euclidean distance and σ² controls the width. This kernel measures similarity between data points in high-dimensional space, producing values between 0 and 1. It is used in t-SNE for computing pairwise affinities between data points, in SVM for nonlinear classification through the kernel trick, and in spectral clustering for constructing graph adjacency matrices. The RBF kernel enables algorithms to model complex relationships without explicitly computing high-dimensional feature representations.

The Radial Basis Function (RBF) kernel is the most widely used and generalized kernel function in machine learning, particularly for Support Vector Machines and neural networks. It transforms n-dimensional input to m-dimensional space by computing the similarity between two points based on their Euclidean distance, using the mathematical formula k(x₁, x₂) = exp(-||x₁-x₂||²/(2σ²)), where σ² is the variance hyperparameter. The kernel value ranges from 0 to 1, with 1 indicating identical points and values decreasing as points become more dissimilar, making it a powerful default choice for non-linear data transformation.

To compute the kernel matrix for RBF kernel, you first select landmark points (centers). For each pair of input points, calculate the squared Euclidean distance between them. Apply the RBF formula: exp(-gamma * distance²). This produces a symmetric matrix where each element represents the similarity between two data points according to the RBF kernel. The diagonal elements (distance zero) will always equal 1 since exp(0) = 1. This kernel matrix is then used by SVM algorithms to transform data into a higher-dimensional space where it may become linearly separable.

This section explains how kernel functions define the covariance structure in Gaussian Processes. A valid kernel must produce positive semi-definite matrices. The kernel encodes the core assumption that similar input points should have similar outputs (high covariance), while dissimilar points should be uncorrelated. The RBF kernel k(X,Z) = exp(-||X-Z||²/σ²) achieves this by measuring Euclidean distance: nearby points yield kernel values near 1 (high correlation), while distant points yield near-zero values (low correlation). This implements the principle that local similarity implies functional similarity. The covariance matrix thus captures the entire correlation structure needed for making predictions.