The Hessian Matrix and the Theory of Extrema
Theorem
A theorem is a proposition that has been rigorously proved, and it is the foundation of mathematical reasoning. A theorem usually states its conditions and its conclusion, and is an important part of any mathematical system.
Suppose the function has continuous partial derivatives of order up to in some neighbourhood of the point . Then for every point in that neighbourhood,
where the second-order term expands as:
This is where the Hessian matrix comes from — it is the coefficient matrix of the second-order term of the Taylor expansion.
Definition
A definition is a precise description of what a concept, a term, or an object means. Understanding definitions is the foundation of learning any field, and every concept has its own exact meaning.
At a stationary point the first-order term vanishes, and the Taylor expansion simplifies to:
where is the Hessian matrix.
The sign of the increment of the function is decided by the quadratic form :
- (positive definite) ⟹ a local minimum
- (negative definite) ⟹ a local maximum
- takes both signs (indefinite) ⟹ a saddle point
- or (semidefinite) ⟹ nothing can be determined
Theorem
A theorem is a proposition that has been rigorously proved, and it is the foundation of mathematical reasoning. A theorem usually states its conditions and its conclusion, and is an important part of any mathematical system.
For an th-order real symmetric matrix , the following conditions are equivalent:
-
Positive definite:
- for every nonzero vector ,
- all the eigenvalues
- all the leading principal minors ()
- there is an invertible matrix with
-
Negative definite: is negative definite ⟺ is positive definite ⟺ the odd-order leading principal minors are and the even-order ones are
-
Indefinite: there are both positive and negative eigenvalues
-
Positive or negative semidefinite: all the eigenvalues are / , with at least one of them zero
Mathematical formula
A formula is a symbolic expression of quantitative relations and structural patterns, and an essential tool for solving mathematical problems. Key formulas should be mastered and applied flexibly.
For the Hessian matrix :
- the first-order leading principal minor
- the second-order leading principal minor
- positive definite ⟺ and ⟹ a local minimum
- negative definite ⟺ and ⟹ a local maximum
- indefinite ⟺ ⟹ a saddle point
- semidefinite ⟺ ⟹ nothing can be determined
Theorem
A theorem is a proposition that has been rigorously proved, and it is the foundation of mathematical reasoning. A theorem usually states its conditions and its conclusion, and is an important part of any mathematical system.
For the constrained extremum problem s.t. (), with the Lagrangian function , define at the stationary point the bordered Hessian matrix:
where is the Jacobian matrix of the constraints and is the Hessian matrix of the Lagrangian function.
The second-order condition for a constrained extremum involves the positive definiteness of on the tangent space of the constraint, which is more complicated than the unconstrained case. At the undergraduate level it is usually judged by the practical meaning or by substitution.
Example 1: Using the Eigenvalues to Decide the Extrema
Find the extrema of , and verify the answer with the eigenvalues.
Approach:
The signs of the eigenvalues of the Hessian at a stationary point decide the type of the extremum directly: all positive gives a local minimum, all negative gives a local maximum, and a mixture gives a saddle point.
Steps:
- and , so the stationary points are , four in all
- : , with eigenvalues , positive definite, a local minimum with
- : , with eigenvalues , negative definite, a local maximum with
- : , with eigenvalues , indefinite, a saddle point
- : , with eigenvalues , indefinite, a saddle point
Answer: the local minimum is , the local maximum is , and the remaining points are saddle points.
Example 2: The Extrema of a Function of Three Variables and the Leading Principal Minors
Find the extrema of .
Approach:
A function of three variables has no discriminant like the two-variable case, so you use the leading principal minors: starting from the first-order minor, watch where the sign changes, and the point where it changes is the extremum point.
Steps:
- , and , so the stationary point is
- The leading principal minors: , and
- The Hessian matrix is positive definite, so is a local minimum point
- The local minimum
Answer: the local minimum is .
Exercise 1
Find the extrema of and classify the stationary points with the eigenvalues of the Hessian.
Approach:
Find the stationary points, compute the Hessian at each of them, and read off the eigenvalues.
Steps:
- and ; setting them to zero gives and , so , hence or
- The stationary points: , and
- At : , with eigenvalues and , both positive, so a local minimum with
- At : the same matrix, so also a local minimum with
- At : , with eigenvalues and , indefinite, so a saddle point with
Answer: the local minima are , and is a saddle point.
Exercise 2
Prove that the Hessian matrix is symmetric when the mixed partial derivatives are continuous.
Approach:
Apply Clairaut’s theorem to the two mixed partial derivatives.
Steps:
- The Hessian matrix is
- A matrix is symmetric exactly when its off-diagonal entries are equal, that is, when
- By Clairaut’s theorem, when the mixed partial derivatives are continuous in a neighbourhood of the point,
- Hence the Hessian matrix is symmetric there.
Answer: the Hessian matrix is symmetric wherever the mixed partial derivatives are continuous.
Symbols Used in This Article
| Symbol | Type | Pronunciation/Explanation | Meaning in This Article |
|---|---|---|---|
| Hessian matrix | Hessian of f | The matrix formed by the second-order partial derivatives | |
| Bordered Hessian | bordered Hessian | The bordered Hessian matrix for a constrained extremum | |
| Leading principal minor | leading principal minor | The -th order leading principal minor | |
| Eigenvalue | eigenvalue | An eigenvalue of the matrix | |
| Quadratic form | quadratic form | The quadratic form defined by the Hessian matrix | |
| Jacobian matrix | Jacobian | The Jacobian matrix of the constraint functions | |
| Remainder | remainder | The remainder of the Taylor expansion |
