Jacobian Matrix Explained: Multivariable Calculus & Change

Linearization is the process of approximating a nonlinear system with a linear system near an equilibrium point. The method involves: (1) Finding equilibrium points where derivatives equal zero, (2) Expanding nonlinear functions using Taylor series around the equilibrium point, (3) Computing the Jacobian matrix of partial derivatives, (4) Forming the linearized system using the Jacobian. This allows analysis of nonlinear systems using well-established linear system techniques. The behavior of solutions near equilibrium points in the nonlinear system is approximately the same as in the linearized system.

Nonlinear systems are more common in practice than linear systems, though most physical systems can be approximated by linear systems. Linearization converts nonlinear systems into something close to linear, working only in narrow ranges around equilibrium points. Nonlinear systems are written in state space form as x_dot = f(x, u) and y = g(x, u), where f and g are arbitrary nonlinear functions. Equilibrium points are points where the system remains constant over time if undisturbed. For continuous-time systems, equilibrium is defined by solving x_dot = 0, meaning the state is constant. For discrete-time systems, the condition is x(k+1) = x(k). The terminology of stable and unstable equilibrium points applies locally around these points.

For non-linear systems, stability analysis uses Taylor series expansion about the equilibrium point. The function is expanded as f(x) = f(x_eq) + ∇f(x_eq) × (x - x_eq) + higher order terms. Since f(x_eq) = 0 at equilibrium, only the first derivative (Jacobian matrix) matters for linearization. The system's stability is determined by the linearized system's stability. If the linearized system is neutrally stable, higher-order terms determine actual stability.

Linearization is a technique for analyzing nonlinear systems by approximating them with linear systems near equilibrium points using Taylor series expansion. For a system x_dot = f(x), the linearized form is Δx_dot = AΔx, where A is the Jacobian matrix (∂f/∂x) evaluated at the equilibrium point. The stability of the equilibrium point is determined by the eigenvalues of A: if all eigenvalues have negative real parts, the equilibrium is stable; if any eigenvalue has a positive real part, it is unstable; if any eigenvalue has zero real part while others are non-positive, stability cannot be determined from linearization alone.

To linearize a non-linear system, we consider the system equations: x_dot = f(x,u) and y = g(x,u). We assume the system is time-invariant and that functions f and g are sufficiently regular (at least differentiable). We identify an equilibrium point (x_eq, u_eq, y_eq) where f(x_eq, u_eq) = 0. We then define deviation variables: Δx = x - x_eq, Δu = u - u_eq, Δy = y - y_eq.

Newton's method generalizes from scalar to vector-valued functions by replacing scalar derivatives with Jacobian matrices. In 1D, x_{n+1} = x_n - f(x_n)/f'(x_n); in nD, x_{n+1} = x_n - D(x_n)^{-1}f(x_n), where D is the Jacobian matrix of partial derivatives. The Jacobian captures how each function changes with respect to each variable, enabling simultaneous solution of n equations with n unknowns.

The Newton-Raphson method for solving systems of n equations with n variables uses the Jacobian matrix (the multivariate analog of the derivative) in the iteration formula: X^(i+1) = X^(i) - J^(-1)(X^(i)) * F(X^(i)), where X^(i) is the current approximation vector, J is the Jacobian matrix evaluated at X^(i), and F(X^(i)) is the function value vector; this method assumes each function is continuous with continuous partial derivatives and starts from an initial approximation vector.

For systems of equations, use Newton-Raphson with the Jacobian matrix J = [[∂f/∂x, ∂f/∂y], [∂g/∂x, ∂g/∂y]]. For the system: (1) f(x,y) = x^2 + y^2 - 25 = 0, (2) g(x,y) = x^2 - y^2 - 7 = 0. Compute partial derivatives: ∂f/∂x = 2x, ∂f/∂y = 2y, ∂g/∂x = 2x, ∂g/∂y = -2y. At (x_0, y_0) = (3, 4): J = [[6, 8], [6, -8]].

The Newton-Raphson method extends to systems of non-linear equations using the Jacobian matrix, which contains first-order partial derivatives of each function with respect to each variable. For n equations and n variables, the Jacobian is an n×n matrix where Jᵢⱼ = ∂fᵢ/∂xⱼ. The multi-dimensional formula becomes xₙ₊₁ = xₙ - J⁻¹(xₙ) × f(xₙ), where J⁻¹ is the inverse Jacobian and f(xₙ) is the function vector. This matrix form generalizes the one-dimensional case, requiring matrix inversion at each iteration. The dimensions must be compatible for the matrix operations to work correctly.

The Jacobian matrix J = [[∂f/∂x, ∂f/∂y], [∂g/∂x, ∂g/∂y]] represents the linear approximation of the nonlinear system. For the example: J = [[2, -2], [1, 1]]. The determinant det(J) = 2(1) - (-2)(1) = 4. For x-update, replace first column with [f, g]^T = [-2, -1]^T to get J_x = [[-2, -2], [-1, 1]], det(J_x) = -4, so x₁ = 1 - (-4)/4 = 2. For y-update, replace second column: J_y = [[2, -2], [1, -1]], det(J_y) = 0, so y₁ = -1 - 0/4 = -1.

The chain rule produces a 'daisy chain' of Jacobians: ∂L/∂w⁽ⁱ⁾ = (∂L/∂z⁽ⁱ⁺¹⁾) × (∂z⁽ⁱ⁺¹⁾/∂a⁽ⁱ⁾) × (∂a⁽ⁱ⁺¹⁾/∂z⁽ⁱ⁺¹⁾) × ... × (∂z⁽ᴸ⁾/∂a⁽ᴸ⁻¹⁾) × (∂a⁽ᴸ⁾/∂z⁽ᴸ⁾) × (∂z⁽ᴸ⁾/∂w⁽ᴸ⁾). Each term is either a weight matrix (∂z/∂a = w) or a diagonal matrix of activation derivatives (∂a/∂z = diag(g'(z))). The error signal δ⁽ⁱ⁺¹⁾ = (∂L/∂z⁽ⁱ⁺¹⁾) is modified by w⁽ⁱ⁺¹⁾ᵀ and g'(z⁽ⁱ⁺¹⁾ as it propagates backwards. The gradient ∂L/∂w⁽ⁱ⁾ = δ⁽ⁱ⁺¹⁾(a⁽ⁱ⁾)ᵀ is an outer product, computationally efficient for implementation. Backpropagation reuses intermediate computations, making it memory-efficient for deep networks.

The multivariate chain rule differs from regular calculus because intermediate values in neural networks are vectors. When computing gradients, we multiply local derivatives of each function in the composition. Jacobian dimensions depend on input/output dimensions: scalar output with vector input produces row vectors, while vector output with vector input produces matrices. This dimensional analysis is crucial for correctly applying the chain rule in neural networks.

The chain rule is a fundamental mathematical principle in calculus that enables efficient computation of derivatives in multi-layered neural networks during backpropagation; it allows us to calculate the derivative of the loss function with respect to each weight by multiplying the derivatives along the path from the output to that weight, enabling systematic weight updates to minimize the loss function.

Backpropagation uses the chain rule from calculus to compute how sensitive a neural network's cost function is to its weights and biases, enabling efficient learning by calculating derivatives that show which parameter adjustments will most effectively reduce the cost; for a single neuron connection, this involves multiplying three derivatives (∂C/∂a × ∂a/∂z × ∂z/∂w), where the sensitivity to weight equals the previous layer's activation times the derivative of the activation function times the cost sensitivity to the output, and this process propagates backward through layers to update all parameters.

Backpropagation computes gradients by traversing the computational graph in reverse (from output to input) and applying the chain rule. Different operations have different gradient propagation rules: addition passes gradients unchanged, multiplication multiplies by the other operand, and sine multiplies by cosine of the input. For branching paths, gradients accumulate: if z = f(g(x) + h(x)), then dz/dx = (df/dg) × (dg/dx) + (df/dh) × (dh/dx). This allows efficient computation of gradients for complex models like deep neural networks.