Hessian Matrix | Multivariable Calculus | Khan Academy

Added:

Computational Bottlenecks and Quasi-Newton Alternatives in High-Dimensional Optimization

While the Hessian matrix is mathematically elegant for organizing second-order derivatives, it faces severe practical limitations in high-dimensional spaces, such as machine learning. Computing and storing an n x n Hessian requires O(n^2) memory, and inverting it costs O(n^3) time, making it computationally intractable for functions with millions of variables. Consequently, applied mathematicians and computer scientists often avoid exact Hessian calculations in favor of first-order methods (like Gradient Descent) or Quasi-Newton methods (such as BFGS or L-BFGS). These alternatives approximate the curvature information using only gradient evaluations, drastically reducing computational overhead while still achieving efficient convergence.

First-order partial derivatives and how to construct the gradient vector of a multivariable function.
Concept and computation of second-order partial derivatives, including mixed partial derivatives and Clairaut's theorem.
Basic linear algebra, particularly matrix representation, vectors, and matrix-vector multiplication.
Single-variable Taylor series, specifically the role of the first and second derivatives in linear and quadratic approximations.
The Multivariable Second Derivative Test to classify local extrema (maxima, minima, and saddle points) of multivariable functions.
Determining matrix definiteness (positive definite, negative definite, indefinite) through the eigenvalues or determinants of the Hessian matrix.
Multivariable Taylor series expansions to higher-degree polynomial approximations.
Optimization algorithms in machine learning and data science, such as Newton's Method, which utilize the inverse of the Hessian matrix.
515.1K views4.6Klikes6:10@khanacademyOriginal Release: 2016-06-16

The Hessian matrix is a square matrix that organizes all the second partial derivatives of a multivariable function, where each entry H_ij represents the second partial derivative obtained by first differentiating with respect to variable j and then with respect to variable i; for most well-behaved functions, the mixed partial derivatives are symmetric (H_ij = H_ji), and this matrix structure extends naturally to functions with any number of variables, making it a powerful tool for expressing quadratic approximations and performing matrix operations in multivariable calculus.