Applications of the Gradient
Theorem
A theorem is a proposition that has been rigorously proved, and it is the foundation of mathematical reasoning. A theorem usually states its conditions and its conclusion, and is an important part of any mathematical system.
- The normal vector of the level curve of a function of two variables at the point is .
- The normal vector of the level surface of a function of three variables at the point is .
- The normal vector of the tangent plane of the surface at the point is .
Application
Given the equation of a level curve or level surface, the gradient directly gives the normal vector, and hence the equations of the tangent line or tangent plane and of the normal line. This is more direct than implicit differentiation.
Mathematical formula
A formula is a symbolic expression of quantitative relations and structural patterns, and an essential tool for solving mathematical problems. Key formulas should be mastered and applied flexibly.
The goal: to find the minimum of the function .
The iteration formula:
where:
- is the iterate of the -th step
- is the learning rate (the step size)
- is the gradient; the minus sign means the update is made in the direction opposite to the gradient (the direction of steepest descent)
The condition for convergence: or .
The Core Idea of Gradient Descent
- The gradient direction is the direction in which the value of the function grows fastest, so the negative gradient direction is the direction in which it decreases fastest.
- At each step you move a small distance along the negative gradient direction, approaching the minimum step by step.
- The choice of the learning rate is crucial: too small and the convergence is slow, too large and it may oscillate or even diverge.
- Gradient descent is one of the most commonly used optimisation algorithms in machine learning (such as the backpropagation of neural networks).
Variants of Gradient Descent
| Method | Iteration formula | Characteristics |
|---|---|---|
| Steepest descent | A fixed learning rate; simple but it may oscillate | |
| Exact line search | the optimal step size is found along the negative gradient direction | optimal at each step, but computationally expensive |
| Momentum | , | accelerates convergence and reduces oscillation |
| Adam | an adaptive learning rate | the most commonly used in machine learning |
Mathematical formula
A formula is a symbolic expression of quantitative relations and structural patterns, and an essential tool for solving mathematical problems. Key formulas should be mastered and applied flexibly.
- The temperature field: gives the distribution of the temperature, points in the direction in which the temperature rises fastest, and the heat flux density is (Fourier’s law); the minus sign says that heat flows from hot to cold.
- The potential field: gives the electric potential, and the electric field strength is ; the field points in the direction in which the potential falls fastest.
- The gravitational field: the gravitational potential , and the gravitational field .
- The concentration field: the concentration , and the diffusive flux (Fick’s law).
The Common Pattern
In physics the direction of a “flux” is usually opposite to the gradient of the potential function — it flows from high potential to low potential. The gradient describes the spatial rate of change of the potential, and is the bridge between the “potential” and the “field”.
Example 1: The Tangent Line and Normal Line of a Level Curve
Find the equations of the tangent line and the normal line of the level curve at the point .
Approach:
The gradient is perpendicular to the level curve, so the direction of the tangent line is orthogonal to the gradient and the direction of the normal line is the gradient itself.
Steps:
- Put , so
- At : , which is the normal vector
- The tangent line: , that is,
- The normal line: , that is,
Answer: the tangent line is , and the normal line is
Example 2: Finding a Minimum with Gradient Descent
Use gradient descent to find the minimum of , starting from the point with a learning rate , and write down the first two iterations.
Approach:
At each step you move a fixed step size along the negative gradient direction; a few more steps make the downward trend of the function value clear.
Steps:
- The starting point , and
- The first step: and
- The second step: and , and
- The value of the function keeps decreasing, approaching the minimum point , and the minimum value is .
Answer: and ; the function value falls from 3 to 0.6688.
Example 3: The Electric Field Strength
Given the potential , find the electric field strength .
Approach:
The electric field strength is by definition the negative gradient of the potential, so you only have to compute the gradient of the potential and negate it.
Steps:
- Put , so
- Similarly and
Answer: , which is the electric field produced by a unit positive charge at the origin (Coulomb’s law).
Exercise 1
Find the equation of the tangent plane of the level surface at the point .
Approach:
The normal vector of the tangent plane of a level surface is the gradient at the point of tangency; write the plane in point-normal form.
Steps:
- Put , so
- At :
- The tangent plane: , that is,
Answer: the equation of the tangent plane is
Exercise 2
The height function of a certain hill is (in units of a hundred metres). At the point , in which direction should a climber ascend to make the least effort (that is, to gain height fastest)? What is the gradient of the slope in that direction?
Approach:
The direction of fastest ascent is the direction of the gradient, and the slope is the magnitude of the gradient.
Steps:
- , and at we have
- The gradient direction is the direction in which the height rises fastest, that is, the direction of least effort for the climber
- The slope (for every unit of horizontal progress, the height rises by about 4.47 hundred metres)
Answer: the climber should ascend in the direction (that is, the direction ), and the slope is about .
Symbols Used in This Article
| Symbol | Type | Pronunciation/Explanation | Meaning in This Article |
|---|---|---|---|
| Iterate | x sub k | The iterate of the -th step of gradient descent | |
| Learning rate | alpha / learning rate | The step size of gradient descent | |
| Electric field | electric field | The vector of the electric field strength | |
| Potential | potential | The potential function | |
| Heat flux | heat flux | The vector of the heat flux density | |
| Thermal conductivity | thermal conductivity | The coefficient in Fourier’s law |
