The Hessian Matrix and the Theory of Extrema

The Taylor Expansion of a Multivariable Function

Theorem1

Suppose the function f(x,y)f(x, y) has continuous partial derivatives of order up to n+1n+1 in some neighbourhood of the point (x0,y0)(x_0, y_0). Then for every point (x0+h,y0+k)(x_0+h, y_0+k) in that neighbourhood,

f(x0+h,y0+k)=f(x0,y0)+(h∂∂x+k∂∂y)f(x0,y0)+12!(h∂∂x+k∂∂y)2f(x0,y0)+⋯+1n!(h∂∂x+k∂∂y)nf(x0,y0)+Rnf(x_0+h, y_0+k) = f(x_0,y_0) + \left(h\frac{\partial}{\partial x} + k\frac{\partial}{\partial y}\right)f(x_0,y_0) + \frac{1}{2!}\left(h\frac{\partial}{\partial x} + k\frac{\partial}{\partial y}\right)^2 f(x_0,y_0) + \cdots + \frac{1}{n!}\left(h\frac{\partial}{\partial x} + k\frac{\partial}{\partial y}\right)^n f(x_0,y_0) + R_n

where the second-order term expands as:

12(h2fxx+2hkfxy+k2fyy)=12(h,k)Hf(hk)\frac{1}{2}(h^2 f_{xx} + 2hk f_{xy} + k^2 f_{yy}) = \frac{1}{2}(h, k) H_f \begin{pmatrix} h \\ k \end{pmatrix}

This is where the Hessian matrix comes from — it is the coefficient matrix of the second-order term of the Taylor expansion.

The Hessian Matrix and the Quadratic Form

The quadratic form and the Hessian matrix

At a stationary point (x0,y0)(x_0, y_0) the first-order term vanishes, and the Taylor expansion simplifies to:

f(x0+h,y0+k)−f(x0,y0)=12(h,k)Hf(hk)+o(ρ2)f(x_0+h, y_0+k) - f(x_0,y_0) = \frac{1}{2}(h, k) H_f \begin{pmatrix} h \\ k \end{pmatrix} + o(\rho^2)

where Hf=(fxxfxyfyxfyy)H_f = \begin{pmatrix} f_{xx} & f_{xy} \\ f_{yx} & f_{yy} \end{pmatrix} is the Hessian matrix.

The sign of the increment of the function is decided by the quadratic form Q(h,k)=(h,k)Hf(h,k)TQ(h,k) = (h,k)H_f(h,k)^T:

  • Q>0Q > 0 (positive definite) ⟹ a local minimum
  • Q<0Q < 0 (negative definite) ⟹ a local maximum
  • QQ takes both signs (indefinite) ⟹ a saddle point
  • Q≥0Q \geq 0 or Q≤0Q \leq 0 (semidefinite) ⟹ nothing can be determined

The Criteria for Positive Definiteness

Theorem2

For an nnth-order real symmetric matrix AA, the following conditions are equivalent:

  1. Positive definite:

    • for every nonzero vector x⃗\vec{x}, x⃗TAx⃗>0\vec{x}^T A \vec{x} > 0
    • all the eigenvalues λi>0\lambda_i > 0
    • all the leading principal minors Dk>0D_k > 0 (k=1,…,nk=1,\ldots,n)
    • there is an invertible matrix CC with A=CTCA = C^T C
  2. Negative definite: AA is negative definite ⟺ −A-A is positive definite ⟺ the odd-order leading principal minors are <0<0 and the even-order ones are >0>0

  3. Indefinite: there are both positive and negative eigenvalues

  4. Positive or negative semidefinite: all the eigenvalues are ≥0\geq 0 / ≤0\leq 0, with at least one of them zero

The discriminant and positive definiteness for a function of two variables

For the 2×22 \times 2 Hessian matrix H=(ABBC)H = \begin{pmatrix} A & B \\ B & C \end{pmatrix}:

  • the first-order leading principal minor D1=AD_1 = A
  • the second-order leading principal minor D2=AC−B2=ΔD_2 = AC - B^2 = \Delta
  • positive definite ⟺ A>0A > 0 and Δ>0\Delta > 0 ⟹ a local minimum
  • negative definite ⟺ A<0A < 0 and Δ>0\Delta > 0 ⟹ a local maximum
  • indefinite ⟺ Δ<0\Delta < 0 ⟹ a saddle point
  • semidefinite ⟺ Δ=0\Delta = 0 ⟹ nothing can be determined

The Second-Order Conditions under Constraints

Theorem3

For the constrained extremum problem min⁡f(x)\min f(x) s.t. gi(x)=0g_i(x)=0 (i=1,…,mi=1,\ldots,m), with the Lagrangian function L=f+∑λigiL = f + \sum \lambda_i g_i, define at the stationary point x∗x^* the bordered Hessian matrix:

HB=(0JJTHL)H_B = \begin{pmatrix} 0 & J \\ J^T & H_L \end{pmatrix}

where JJ is the Jacobian matrix of the constraints and HLH_L is the Hessian matrix of the Lagrangian function.

The second-order condition for a constrained extremum involves the positive definiteness of HLH_L on the tangent space of the constraint, which is more complicated than the unconstrained case. At the undergraduate level it is usually judged by the practical meaning or by substitution.

Worked Examples

Example 1: Using the Eigenvalues to Decide the Extrema

Find the extrema of f(x,y)=x3+y3−3x−3yf(x,y) = x^3 + y^3 - 3x - 3y, and verify the answer with the eigenvalues.

the Hessian matrixeigenvaluesan extremumpositive definiteness

Approach:

The signs of the eigenvalues of the Hessian at a stationary point decide the type of the extremum directly: all positive gives a local minimum, all negative gives a local maximum, and a mixture gives a saddle point.

Steps:

  1. fx=3x2−3f_x = 3x^2-3 and fy=3y2−3f_y = 3y^2-3, so the stationary points are (±1,±1)(\pm1, \pm1), four in all
  2. H=(6x006y)H = \begin{pmatrix} 6x & 0 \\ 0 & 6y \end{pmatrix}
  3. (1,1)(1,1): H=(6006)H=\begin{pmatrix}6&0\\0&6\end{pmatrix}, with eigenvalues 6,6>06,6>0, positive definite, a local minimum with f=−4f=-4
  4. (−1,−1)(-1,-1): H=(−600−6)H=\begin{pmatrix}-6&0\\0&-6\end{pmatrix}, with eigenvalues −6,−6<0-6,-6<0, negative definite, a local maximum with f=4f=4
  5. (1,−1)(1,-1): H=(600−6)H=\begin{pmatrix}6&0\\0&-6\end{pmatrix}, with eigenvalues 6,−66,-6, indefinite, a saddle point
  6. (−1,1)(-1,1): H=(−6006)H=\begin{pmatrix}-6&0\\0&6\end{pmatrix}, with eigenvalues −6,6-6,6, indefinite, a saddle point

Answer: the local minimum is f(1,1)=−4f(1,1)=-4, the local maximum is f(−1,−1)=4f(-1,-1)=4, and the remaining points are saddle points.

Example 2: The Extrema of a Function of Three Variables and the Leading Principal Minors

Find the extrema of f(x,y,z)=x2+y2+z2+2x+4y−6zf(x,y,z) = x^2 + y^2 + z^2 + 2x + 4y - 6z.

the Hessian matrixleading principal minorsa function of three variablesan extremum

Approach:

A function of three variables has no discriminant like the two-variable case, so you use the leading principal minors: starting from the first-order minor, watch where the sign changes, and the point where it changes is the extremum point.

Steps:

  1. fx=2x+2f_x=2x+2, fy=2y+4f_y=2y+4 and fz=2z−6f_z=2z-6, so the stationary point is (−1,−2,3)(-1,-2,3)
  2. H=(200020002)H = \begin{pmatrix} 2 & 0 & 0 \\ 0 & 2 & 0 \\ 0 & 0 & 2 \end{pmatrix}
  3. The leading principal minors: D1=2>0D_1=2>0, D2=∣2002∣=4>0D_2=\begin{vmatrix}2&0\\0&2\end{vmatrix}=4>0 and D3=8>0D_3=8>0
  4. The Hessian matrix is positive definite, so (−1,−2,3)(-1,-2,3) is a local minimum point
  5. The local minimum f(−1,−2,3)=1+4+9−2−8−18=−14f(-1,-2,3) = 1+4+9-2-8-18 = -14

Answer: the local minimum is −14-14.


Exercises

Exercise 1

Find the extrema of f(x,y)=x4+y4−4xyf(x,y) = x^4 + y^4 - 4xy and classify the stationary points with the eigenvalues of the Hessian.

the Hessian matrixeigenvaluesan extremuma classification of the stationary points

Approach:

Find the stationary points, compute the Hessian at each of them, and read off the eigenvalues.

Steps:

  1. fx=4x3−4yf_x = 4x^3-4y and fy=4y3−4xf_y = 4y^3-4x; setting them to zero gives y=x3y=x^3 and x=y3x=y^3, so x=x9x=x^9, hence x=0x=0 or x=±1x=\pm1
  2. The stationary points: (0,0)(0,0), (1,1)(1,1) and (−1,−1)(-1,-1)
  3. H=(12x2−4−412y2)H = \begin{pmatrix} 12x^2 & -4 \\ -4 & 12y^2 \end{pmatrix}
  4. At (1,1)(1,1): H=(12−4−412)H=\begin{pmatrix}12&-4\\-4&12\end{pmatrix}, with eigenvalues 12−4=812-4=8 and 12+4=1612+4=16, both positive, so a local minimum with f=1+1−4=−2f=1+1-4=-2
  5. At (−1,−1)(-1,-1): the same matrix, so also a local minimum with f=−2f=-2
  6. At (0,0)(0,0): H=(0−4−40)H=\begin{pmatrix}0&-4\\-4&0\end{pmatrix}, with eigenvalues −4-4 and 44, indefinite, so a saddle point with f=0f=0

Answer: the local minima are f(1,1)=f(−1,−1)=−2f(1,1)=f(-1,-1)=-2, and (0,0)(0,0) is a saddle point.

Exercise 2

Prove that the Hessian matrix is symmetric when the mixed partial derivatives are continuous.

the Hessian matrixClairaut's theorema proof

Approach:

Apply Clairaut’s theorem to the two mixed partial derivatives.

Steps:

  1. The Hessian matrix is Hf=(fxxfxyfyxfyy)H_f = \begin{pmatrix} f_{xx} & f_{xy} \\ f_{yx} & f_{yy} \end{pmatrix}
  2. A matrix is symmetric exactly when its off-diagonal entries are equal, that is, when fxy=fyxf_{xy} = f_{yx}
  3. By Clairaut’s theorem, when the mixed partial derivatives are continuous in a neighbourhood of the point, fxy=fyxf_{xy} = f_{yx}
  4. Hence the Hessian matrix is symmetric there.

Answer: the Hessian matrix is symmetric wherever the mixed partial derivatives are continuous.


Summary

Symbols Used in This Article

SymbolTypePronunciation/ExplanationMeaning in This Article
HfH_fHessian matrixHessian of fThe matrix formed by the second-order partial derivatives
HBH_BBordered Hessianbordered HessianThe bordered Hessian matrix for a constrained extremum
DkD_kLeading principal minorleading principal minorThe kk-th order leading principal minor
λi\lambda_iEigenvalueeigenvalueAn eigenvalue of the matrix
QQQuadratic formquadratic formThe quadratic form defined by the Hessian matrix
JJJacobian matrixJacobianThe Jacobian matrix of the constraint functions
RnR_nRemainderremainderThe remainder of the Taylor expansion

Found an issue with this content?

Tell us what could be improved.