Orthogonal Matrices & Gram-Schmidt | Linear Algebra

Added:

Numerical Instability and Householder QR Decomposition

While the classical Gram-Schmidt process is a foundational theoretical tool for constructing orthonormal bases, it suffers from severe numerical instability when implemented in computer floating-point arithmetic. As the dimension of the vector space grows, rounding errors accumulate, leading to a catastrophic loss of orthogonality among the resulting vectors. To address this limitation, numerical analysts and computational scientists prefer alternative methods. The Modified Gram-Schmidt (MGS) algorithm improves stability but does not fully resolve the issue. Instead, Householder reflections (or Givens rotations) are widely used to compute the QR decomposition. These alternative methods are backward stable and preserve orthogonality much more reliably in practical computational environments. Therefore, while Gram-Schmidt is pedagogically valuable, it is often bypassed in applied linear algebra in favor of Householder-based orthogonalization.

Vector dot products, inner product spaces, and the geometric concept of orthogonality (perpendicularity).
Linear independence, vector spans, and the definition of a basis for a vector space.
Vector projection, specifically how to project one vector onto another or onto a subspace.
Basic matrix algebra, including matrix multiplication, transposes, and the identity matrix.
QR Decomposition, which factorizes a matrix into an orthogonal matrix Q and an upper triangular matrix R.
The Least Squares Method for solving overdetermined systems of equations efficiently using orthogonal bases.
Orthogonal diagonalization, symmetric matrices, and the Spectral Theorem.
Singular Value Decomposition (SVD) and its application in dimensionality reduction (such as PCA).
280.2K views3.8Klikes49:09@mitocwOriginal Release: 2019-09-24

An orthogonal matrix has orthonormal columns where Q^TQ = I, making calculations like projections and least squares solutions dramatically simpler (e.g., projection matrix becomes QQ^T and normal equations reduce to x̂ = Q^TB); the Gram-Schmidt process transforms any set of independent vectors into an orthonormal basis by iteratively subtracting projections onto previously constructed orthogonal vectors, with the relationship A = QR connecting the original matrix A to the orthonormal matrix Q via an upper triangular matrix R.