Applications of the Gradient

The Normal Vectors of Level Curves and Level Surfaces

Theorem1
  1. The normal vector of the level curve f(x,y)=cf(x,y) = c of a function of two variables z=f(x,y)z = f(x,y) at the point (x0,y0)(x_0,y_0) is ∇f(x0,y0)\nabla f(x_0,y_0).
  2. The normal vector of the level surface f(x,y,z)=cf(x,y,z) = c of a function of three variables u=f(x,y,z)u = f(x,y,z) at the point (x0,y0,z0)(x_0,y_0,z_0) is ∇f(x0,y0,z0)\nabla f(x_0,y_0,z_0).
  3. The normal vector of the tangent plane of the surface F(x,y,z)=0F(x,y,z) = 0 at the point M0M_0 is ∇F(M0)\nabla F(M_0).

The Method of Steepest Descent (Gradient Descent)

The basic iteration of gradient descent

The goal: to find the minimum of the function f(x⃗)f(\vec{x}).

The iteration formula:

x⃗k+1=x⃗k−αk∇f(x⃗k)\vec{x}_{k+1} = \vec{x}_k - \alpha_k \nabla f(\vec{x}_k)

where:

  • x⃗k\vec{x}_k is the iterate of the kk-th step
  • αk>0\alpha_k > 0 is the learning rate (the step size)
  • ∇f(x⃗k)\nabla f(\vec{x}_k) is the gradient; the minus sign means the update is made in the direction opposite to the gradient (the direction of steepest descent)

The condition for convergence: ∣∇f(x⃗k)∣<ε|\nabla f(\vec{x}_k)| < \varepsilon or ∣f(x⃗k+1)−f(x⃗k)∣<ε|f(\vec{x}_{k+1}) - f(\vec{x}_k)| < \varepsilon.

Variants of Gradient Descent

MethodIteration formulaCharacteristics
Steepest descentx⃗k+1=x⃗k−α∇f(x⃗k)\vec{x}_{k+1} = \vec{x}_k - \alpha \nabla f(\vec{x}_k)A fixed learning rate; simple but it may oscillate
Exact line searchthe optimal step size is found along the negative gradient directionoptimal at each step, but computationally expensive
Momentumvk+1=βvk−α∇fv_{k+1} = \beta v_k - \alpha \nabla f, x⃗k+1=x⃗k+vk+1\vec{x}_{k+1} = \vec{x}_k + v_{k+1}accelerates convergence and reduces oscillation
Adaman adaptive learning ratethe most commonly used in machine learning

The Gradient in Physical Fields

The gradient in physics
  1. The temperature field: T(x,y,z)T(x,y,z) gives the distribution of the temperature, ∇T\nabla T points in the direction in which the temperature rises fastest, and the heat flux density is q⃗=−k∇T\vec{q} = -k\nabla T (Fourier’s law); the minus sign says that heat flows from hot to cold.
  2. The potential field: φ(x,y,z)\varphi(x,y,z) gives the electric potential, and the electric field strength is E⃗=−∇φ\vec{E} = -\nabla \varphi; the field points in the direction in which the potential falls fastest.
  3. The gravitational field: the gravitational potential VV, and the gravitational field g⃗=−∇V\vec{g} = -\nabla V.
  4. The concentration field: the concentration c(x,y,z)c(x,y,z), and the diffusive flux J⃗=−D∇c\vec{J} = -D\nabla c (Fick’s law).

Worked Examples

Example 1: The Tangent Line and Normal Line of a Level Curve

Find the equations of the tangent line and the normal line of the level curve x2+2y2=3x^2 + 2y^2 = 3 at the point (1,1)(1, 1).

level curvethe gradientthe tangent linethe normal line

Approach:

The gradient is perpendicular to the level curve, so the direction of the tangent line is orthogonal to the gradient and the direction of the normal line is the gradient itself.

Steps:

  1. Put f(x,y)=x2+2y2f(x,y) = x^2 + 2y^2, so ∇f=(2x,4y)\nabla f = (2x, 4y)
  2. At (1,1)(1,1): ∇f=(2,4)\nabla f = (2, 4), which is the normal vector
  3. The tangent line: 2(x−1)+4(y−1)=02(x-1) + 4(y-1) = 0, that is, x+2y−3=0x + 2y - 3 = 0
  4. The normal line: x−12=y−14\frac{x-1}{2} = \frac{y-1}{4}, that is, 2x−y−1=02x - y - 1 = 0

Answer: the tangent line is x+2y−3=0x+2y-3=0, and the normal line is 2x−y−1=02x-y-1=0

Example 2: Finding a Minimum with Gradient Descent

Use gradient descent to find the minimum of f(x,y)=x2+2y2f(x,y) = x^2 + 2y^2, starting from the point (1,1)(1,1) with a learning rate α=0.1\alpha = 0.1, and write down the first two iterations.

gradient descentoptimisationiteration

Approach:

At each step you move a fixed step size along the negative gradient direction; a few more steps make the downward trend of the function value clear.

Steps:

  1. ∇f=(2x,4y)\nabla f = (2x, 4y)
  2. The starting point x⃗0=(1,1)\vec{x}_0 = (1,1), ∇f(x⃗0)=(2,4)\nabla f(\vec{x}_0) = (2,4) and f(x⃗0)=3f(\vec{x}_0) = 3
  3. The first step: x⃗1=(1,1)−0.1(2,4)=(0.8,0.6)\vec{x}_1 = (1,1) - 0.1(2,4) = (0.8, 0.6) and f(x⃗1)=0.64+0.72=1.36f(\vec{x}_1) = 0.64 + 0.72 = 1.36
  4. The second step: ∇f(x⃗1)=(1.6,2.4)\nabla f(\vec{x}_1) = (1.6, 2.4) and x⃗2=(0.8,0.6)−0.1(1.6,2.4)=(0.64,0.36)\vec{x}_2 = (0.8,0.6) - 0.1(1.6,2.4) = (0.64, 0.36), and f(x⃗2)=0.4096+0.2592=0.6688f(\vec{x}_2) = 0.4096 + 0.2592 = 0.6688
  5. The value of the function keeps decreasing, approaching the minimum point (0,0)(0,0), and the minimum value is 00.

Answer: x⃗1=(0.8,0.6)\vec{x}_1=(0.8,0.6) and x⃗2=(0.64,0.36)\vec{x}_2=(0.64,0.36); the function value falls from 3 to 0.6688.

Example 3: The Electric Field Strength

Given the potential φ(x,y,z)=1x2+y2+z2\varphi(x,y,z) = \frac{1}{\sqrt{x^2+y^2+z^2}}, find the electric field strength E⃗=−∇φ\vec{E} = -\nabla \varphi.

the gradientthe electric potentialthe electric field strengtha physical application

Approach:

The electric field strength is by definition the negative gradient of the potential, so you only have to compute the gradient of the potential and negate it.

Steps:

  1. Put r=x2+y2+z2r = \sqrt{x^2+y^2+z^2}, so φ=1r\varphi = \frac{1}{r}
  2. ∂φ∂x=−1r2⋅xr=−xr3\frac{\partial \varphi}{\partial x} = -\frac{1}{r^2} \cdot \frac{x}{r} = -\frac{x}{r^3}
  3. Similarly ∂φ∂y=−yr3\frac{\partial \varphi}{\partial y} = -\frac{y}{r^3} and ∂φ∂z=−zr3\frac{\partial \varphi}{\partial z} = -\frac{z}{r^3}
  4. ∇φ=−1r3(x,y,z)=−r⃗r3\nabla \varphi = -\frac{1}{r^3}(x, y, z) = -\frac{\vec{r}}{r^3}
  5. E⃗=−∇φ=r⃗r3=(x,y,z)(x2+y2+z2)3/2\vec{E} = -\nabla \varphi = \frac{\vec{r}}{r^3} = \frac{(x,y,z)}{(x^2+y^2+z^2)^{3/2}}

Answer: E⃗=(x,y,z)(x2+y2+z2)3/2\vec{E} = \frac{(x,y,z)}{(x^2+y^2+z^2)^{3/2}}, which is the electric field produced by a unit positive charge at the origin (Coulomb’s law).


Exercises

Exercise 1

Find the equation of the tangent plane of the level surface x2+y2+z2=9x^2 + y^2 + z^2 = 9 at the point (2,2,1)(2, 2, 1).

level surfacethe gradientthe tangent plane

Approach:

The normal vector of the tangent plane of a level surface is the gradient at the point of tangency; write the plane in point-normal form.

Steps:

  1. Put F=x2+y2+z2F = x^2+y^2+z^2, so ∇F=(2x,2y,2z)\nabla F = (2x,2y,2z)
  2. At (2,2,1)(2,2,1): ∇F=(4,4,2)\nabla F = (4,4,2)
  3. The tangent plane: 4(x−2)+4(y−2)+2(z−1)=04(x-2)+4(y-2)+2(z-1)=0, that is, 2x+2y+z−9=02x+2y+z-9=0

Answer: the equation of the tangent plane is 2x+2y+z−9=02x+2y+z-9=0

Exercise 2

The height function of a certain hill is h(x,y)=10−x2−2y2h(x,y) = 10 - x^2 - 2y^2 (in units of a hundred metres). At the point (1,1)(1,1), in which direction should a climber ascend to make the least effort (that is, to gain height fastest)? What is the gradient of the slope in that direction?

the gradientsteepest ascentthe slopean application

Approach:

The direction of fastest ascent is the direction of the gradient, and the slope is the magnitude of the gradient.

Steps:

  1. ∇h=(−2x,−4y)\nabla h = (-2x, -4y), and at (1,1)(1,1) we have ∇h=(−2,−4)\nabla h = (-2, -4)
  2. The gradient direction (−2,−4)(-2,-4) is the direction in which the height rises fastest, that is, the direction of least effort for the climber
  3. The slope =∣∇h∣=4+16=25≈4.47= |\nabla h| = \sqrt{4+16} = 2\sqrt{5} \approx 4.47 (for every unit of horizontal progress, the height rises by about 4.47 hundred metres)

Answer: the climber should ascend in the direction (−2,−4)(-2,-4) (that is, the direction (−1,−2)(-1,-2)), and the slope is about 252\sqrt{5}.


Summary

Symbols Used in This Article

SymbolTypePronunciation/ExplanationMeaning in This Article
x⃗k\vec{x}_kIteratex sub kThe iterate of the kk-th step of gradient descent
α\alphaLearning ratealpha / learning rateThe step size of gradient descent
E⃗\vec{E}Electric fieldelectric fieldThe vector of the electric field strength
φ\varphiPotentialpotentialThe potential function
q⃗\vec{q}Heat fluxheat fluxThe vector of the heat flux density
kkThermal conductivitythermal conductivityThe coefficient in Fourier’s law

Found an issue with this content?

Tell us what could be improved.