Solving Linear Systems by Elimination | MIT Linear Algebra

LU decomposition factors a square matrix A into a lower triangular matrix L and an upper triangular matrix U (A = LU). The process follows Naive Gaussian method's forward elimination steps: first eliminate elements below the first pivot using multipliers, recording these values in L; then eliminate elements below the second pivot using new multipliers. Each multiplier corresponds to its position in L where elimination occurred. The resulting matrix becomes upper triangular U, while L contains all multipliers used. This decomposition enables efficient solution of linear systems through forward and backward substitution.

LU decomposition factorizes a matrix A into lower triangular matrix L and upper triangular matrix U such that A = LU. L has 1s on its diagonal. To perform LU decomposition: apply row operations to transform A into upper triangular matrix U, recording multipliers used as entries in L. For matrix [[1,1,2],[1,2,3],[2,1,4]], L = [[1,0,0],[1,1,0],[2,-1,1]] and U = [[1,1,2],[0,1,1],[0,0,1]]. The entry l32 = -1 and u33 = 1.

LU factorization decomposes a matrix into the product of a lower triangular matrix L and an upper triangular matrix U (K = LU). This factorization represents the elimination process in matrix form, where E (the elimination matrix) transforms K into U. The elimination matrix E contains the multipliers used during row operations. By moving E to the other side (multiplying by E inverse), we obtain the factorization K = LU. This decomposition is valuable because once factored, solving multiple systems with the same coefficient matrix becomes much more efficient. The lower triangular matrix L contains the multipliers from elimination, while U contains the resulting upper triangular form.

LU Decomposition factors a matrix A into the product of a lower triangular matrix L and an upper triangular matrix U (A = LU). The process involves: (1) Starting with the identity matrix for L, (2) Applying row operations to transform A into U while simultaneously recording multipliers in L, (3) Ensuring L has 1s on the diagonal and U has zeros below the diagonal. For a 3x3 matrix, this involves operations like R2 = R2 - l21×R1, R3 = R3 - l31×R1, then R3 = R3 - l32×R2. This decomposition is useful for solving linear systems and matrix inversion.

LU decomposition factors a square invertible matrix A into the product of a lower triangular matrix L and an upper triangular matrix U. The process uses Gaussian elimination: select a pivot, eliminate entries below it by subtracting multiples of the pivot row. For 2x2 matrices, multiplying by an elimination matrix E21 eliminates the (2,1) entry, producing an upper triangular U. The L matrix is the inverse of the product of all elimination matrices, with 1s on the diagonal and negated multipliers below. For 3x3 matrices, multiple elimination steps are needed with different pivots. The L matrix is constructed by collecting all multipliers used during elimination: each entry L_ij (i > j) contains the multiplier used to eliminate entry (i,j). The diagonal entries of L are all 1. The U matrix contains pivot values on its diagonal.

Elimination fails when a zero appears in a pivot position and cannot be resolved by row exchanges. There are two types of failure: temporary failure, where a zero pivot can be fixed by exchanging rows to bring a non-zero value into the pivot position; and complete failure, where no non-zero value exists below the problematic zero, meaning the matrix is singular (non-invertible). A matrix is singular if it does not have full rank, which means its determinant is zero and it lacks a complete set of pivots. The ability to identify these failure conditions is crucial for understanding when a system of equations has no unique solution.

When a zero pivot is encountered during elimination, row exchange can be used to fix the situation by swapping the current row with a row below it that has a non-zero element in the pivot column. A system is singular when elimination produces fewer pivots than the number of equations. For a 3×3 system, if only two pivots are obtained (one row has all zeros below the pivot), the system is singular. The system is not invertible, meaning you cannot solve it by multiplying by the inverse of the coefficient matrix.

When Gaussian elimination encounters a zero pivot, the algorithm must find a non-zero element in the same column below the current position to use as a pivot. This requires exchanging rows using a permutation matrix. The video demonstrates this with a 4×5 matrix where the first pivot is zero, requiring a row swap between rows 1 and 2. The permutation matrix is created by swapping corresponding rows in the identity matrix. An important property is that squaring the permutation matrix yields the identity matrix, confirming that applying the same row exchange twice restores the original arrangement. This systematic approach ensures numerical stability and allows the elimination process to continue successfully.

Pivoting is used in Gaussian elimination when the pivot element is zero or close to zero. When this occurs, rows must be exchanged to bring a non-zero element to the pivot position. Permutation matrices represent row exchanges, being identity matrices with two rows swapped. For example, swapping rows 1 and 2 results in a matrix with 1s at positions (2,1) and (1,2). Multiple row exchanges can be combined by multiplying multiple permutation matrices. The order of multiplication matters because matrix multiplication is not commutative.

In Gaussian elimination, unknowns are often omitted and the system is represented as an augmented matrix [A|b], which contains the coefficient matrix with an additional column for right-hand side values. Gaussian elimination cannot always be completed for singular systems. When a zero pivot is encountered, row exchange (pivoting) can be used to bring a non-zero element to the pivot position.

Gauss-Jordan elimination extends Gaussian elimination by continuing row operations until the matrix reaches reduced row echelon form (RREF). In RREF: (1) The leading entry in each non-zero row is 1, (2) Each leading 1 is the only non-zero entry in its column, (3) Zero rows are at the bottom. This method directly reveals the solution without back substitution.

For a 2×2 matrix, Gauss-Jordan elimination involves: (1) Augment A with I, (2) Perform elimination steps to convert A to upper triangular form, (3) Continue elimination upwards to convert to reduced row echelon form (identity), (4) The augmented part becomes A⁻¹. For example, with matrix [[1,3],[2,7]], the inverse is [[7,-3],[-2,1]].

The Gauss-Jordan elimination method is an extension of Gaussian elimination that transforms an augmented matrix [A|I] into [I|A⁻¹] by performing row operations to convert the left side into the identity matrix, thereby obtaining the inverse of matrix A on the right side. This method is particularly useful for finding matrix inverses efficiently, as it allows solving for all columns of the inverse simultaneously rather than solving separate systems for each column.

Row reduced echelon form (RREF) is more restrictive than row echelon form: (1) matrix must be in row echelon form, (2) all pivots must be 1, (3) all entries above and below pivots in pivotal columns must be zero. Gauss-Jordan elimination uses the same three row operations but continues reduction to achieve RREF. The process involves: making each pivot equal to 1, creating zeros below pivots, then creating zeros above pivots. This method allows direct reading of variable values without back-substitution, as each pivot corresponds to a variable.

To find the inverse of a matrix A, augment A with the identity matrix [A|I] and perform row operations to transform A into I. The right side will then be A⁻¹. For the matrix [[1,1,1],[1,3,1],[1,1,2]], augment with I: [[1,1,1|1,0,0],[1,3,1|0,1,0],[1,1,2|0,0,1]]. Subtract row 1 from rows 2 and 3: [[1,1,1|1,0,0],[0,2,0|-1,1,0],[0,0,1|-1,0,1]]. Divide row 2 by 2: [[1,1,1|1,0,0],[0,1,0|-1/2,1/2,0],[0,0,1|-1,0,1]]. Subtract row 2 from row 1: [[1,0,1|3/2,-1/2,0],[0,1,0|-1/2,1/2,0],[0,0,1|-1,0,1]]. Subtract row 3 from row 1: [[1,0,0|5/2,-1/2,1],[0,1,0|-1/2,1/2,0],[0,0,1|-1,0,1]]. So A⁻¹ = [[5/2,-1/2,1],[-1/2,1/2,0],[-1,0,1]].

The range (or column space) of a matrix A is the set of all vectors that can be expressed as Ax for some x, which equals the span of the columns of A. The null space (or kernel) of A is the set of all vectors x such that Ax = 0. These concepts are fundamental for solving linear equation systems Ax = b: a solution exists if and only if b is in the range of A, and if a solution exists, the number of solutions depends on the null space—unique solution if null space is trivial, infinitely many solutions if null space is non-trivial.

This section develops the complete solution methodology for linear systems. After establishing solvability, the instructor presents the algorithm: first find a particular solution by setting free variables to zero and solving for pivots through back substitution. Then recognize that the complete solution is the particular solution plus any vector from the null space. This structure works because AX_particular = B and AX_null = 0, so their sum satisfies A(X_particular + X_null) = B. The geometric interpretation shows solutions form a translated subspace (a plane in higher dimensions) rather than passing through the origin.

The complete solution to Ax = b consists of a particular solution plus any vector from the null space of A. If the complete solution is given as x_p + x_n, where x_p is a particular solution and x_n is in the null space, then the dimension of the null space is equal to the number of free variables in the solution. In the example, two independent vectors in the null space indicate the nullity is 2.

Every matrix A has two fundamental subspaces: the column space (Col(A)), which is the set of all output vectors B where Ax = B for some x, equivalent to the span of the columns of A and living in the codomain R^m; and the null space (Null(A)), which is the set of all input vectors x where Ax = 0, representing vectors 'killed off' by the transformation and living in the domain R^n. Both are subspaces: the column space is a subspace by definition of span, while the null space is a subspace because it satisfies the three subspace conditions (contains zero vector, closed under scalar multiplication, and closed under addition) due to the linearity of matrix multiplication.