The Gradient
Definition
A definition is a precise description of what a concept, a term, or an object means. Understanding definitions is the foundation of learning any field, and every concept has its own exact meaning.
Suppose the function has first-order partial derivatives at the point . Then the vector
is called the gradient of the function at the point , written or .
Here is called the nabla operator (the del operator).
For a function of three variables , the gradient is:
The magnitude of the gradient is:
Theorem
A theorem is a proposition that has been rigorously proved, and it is the foundation of mathematical reasoning. A theorem usually states its conditions and its conclusion, and is an important part of any mathematical system.
Suppose the function is differentiable at the point , and let be any unit direction vector. Then the directional derivative can be written as the inner product of the gradient with the direction vector:
where is the angle between the gradient and the direction .
The Core Properties of the Gradient
- The gradient direction is the direction in which the value of the function grows fastest (the direction of steepest ascent); here , and the directional derivative attains its maximum value .
- The direction opposite to the gradient is the direction in which the value of the function falls fastest (the direction of steepest descent); here the directional derivative attains its minimum value .
- The magnitude of the gradient is the maximum directional derivative, that is, the greatest rate of change of the function at that point.
- When the direction is perpendicular to the gradient, the directional derivative is 0, and the value of the function does not change along that direction.
This is the mathematical foundation of gradient descent — updating the parameters in the direction opposite to the gradient decreases the value of the objective function as fast as possible.
Mathematical formula
A formula is a symbolic expression of quantitative relations and structural patterns, and an essential tool for solving mathematical problems. Key formulas should be mastered and applied flexibly.
- For a function of two variables: the gradient is the normal vector of the level curve at the point , pointing in the direction in which the value of the function increases.
- For a function of three variables: the gradient is the normal vector of the level surface at that point, pointing in the direction in which the value of the function increases.
An Intuitive Understanding
Imagine a hill; the contour lines are the level curves. The gradient direction is perpendicular to the contour lines and points towards the summit (the direction in which the value of the function increases), and climbing in this direction is the steepest. The magnitude of the gradient measures how steep the hill is — the larger the magnitude, the steeper the hill.
Theorem
A theorem is a proposition that has been rigorously proved, and it is the foundation of mathematical reasoning. A theorem usually states its conditions and its conclusion, and is an important part of any mathematical system.
- Linearity:
- The product rule:
- The quotient rule: ()
- Composite functions:
Example 1: Computing the Gradient
Find the gradient of at the point , and find the directional derivative along the gradient direction.
Approach:
The gradient is the vector formed by the partial derivatives; along the gradient direction the function changes fastest, and along the negative gradient direction it falls fastest.
Steps:
- , and at we have
- The unit vector in the gradient direction:
- The directional derivative along the gradient direction =
- Checking:
Answer: the gradient is , and the directional derivative along the gradient direction is (the maximum).
Example 2: The Gradient of a Function of Three Variables
Find the magnitude and the direction cosines of the gradient of at the point .
Approach:
First compute the three partial derivatives and substitute the given point; the magnitude of the gradient and the direction cosines both follow directly from these partial derivatives.
Steps:
- , and
- At : , and
- and
- The direction cosines: , and
Answer: the magnitude of the gradient is , and the direction cosines are .
Example 3: The Normal Vector of a Level Curve
Find the gradient of the function at the point , and explain its relation to the level curve.
Approach:
The components of the gradient are exactly the two partial derivatives of the implicit equation of the level curve, so the gradient direction is the normal to the level curve.
Steps:
- and
- At :
- The level curve through is (an ellipse)
- The gradient is the normal vector of that ellipse at , pointing in the direction in which the value of the function increases (outside the ellipse).
Answer: the gradient is , and it is the normal vector of the level curve (the ellipse) at that point.
Exercise 1
Find the gradient of at the point .
Approach:
Compute the two partial derivatives and substitute the point.
Steps:
- and
- At :
Answer: the gradient is
Exercise 2
Find the direction in which the function changes fastest at the point , and find the greatest rate of change.
Approach:
The direction of fastest change is the direction of the gradient, and the greatest rate of change is the magnitude of the gradient.
Steps:
- , and
- At :
- The direction of fastest change is the gradient direction
- The greatest rate of change =
Answer: the direction of fastest change is , and the greatest rate of change is .
Symbols Used in This Article
| Symbol | Type | Pronunciation/Explanation | Meaning in This Article |
|---|---|---|---|
| Gradient | gradient of f / nabla f | The gradient of the function | |
| Gradient | grad f | Another notation for the gradient | |
| Nabla operator | nabla / del | The differential operator | |
| Magnitude of the gradient | magnitude of gradient | The length of the gradient vector | |
| Angle | theta | The angle between the gradient and the direction vector |
