The Dot Product and Duality
The dot product of two vectors of the same dimension multiplies the corresponding components and adds them up (the result is a number):
Example: v = (2, 3), w = (4, 1)
v·w = 2×4 + 3×1 = 8 + 3 = 11
Note: the inputs of the dot product are two vectors, and the output is a scalar (a number).
The dot product also has a completely equivalent geometric interpretation, best understood through projection:
where is the angle between the two vectors.
Intuition: project onto the direction of (or the other way round), and the dot product is “the length of × the length of the projection of in the direction of ”.
| Angle θ | cos θ | Sign of v·w | Meaning |
|---|---|---|---|
| θ < 90° | positive | positive | The two vectors point roughly the same way |
| θ = 90° | 0 | 0 | The two vectors are perpendicular (orthogonal) |
| θ > 90° | negative | negative | The two vectors point roughly opposite ways |
An important conclusion: (perpendicular ⟺ the dot product is zero).
Verifying perpendicularity:
v = (1, 0), w = (0, 1)
v·w = 1×0 + 0×1 = 0 ✓ the two axes are mutually perpendicular
Now we can reveal a deep fact: the dot product and a linear transformation are the same thing.
A linear transformation from “vectors to numbers”
Consider a linear transformation (the output is a scalar). Its matrix is a 1×2 row vector:
The key insight: the result of this transformation is exactly the dot product of a vector with :
The Core of Duality
Every linear transformation that maps a vector to a number corresponds to a unique vector — that vector is the row of the transformation’s matrix. Taking the dot product with it is equivalent to performing the transformation.
Geometric meaning: any “vector → number” linear transformation can be understood as “projecting the vector onto some direction and then scaling”. That direction is the direction of the unique vector.
Why this matters for the dot product
This explains why the dot product is symmetric () and why the direction of the projection can be swapped: projection is essentially a linear transformation, and a linear transformation can be encoded by a vector. The dot product and a linear transformation are two sides of the same coin.
A measure of similarity: the larger the dot product of two vectors, the more nearly they point the same way (the more similar they are).
| Application | How it is used |
|---|---|
| Cosine similarity | , measuring how well the directions agree |
| Recommender systems | the dot product of a user vector and an item vector = the degree of match |
| Neural networks | each output neuron of a linear layer is a dot product of the input with a weight vector |
| The attention mechanism | the dot product of a Query and a Key determines the strength of attention |
An example from the attention mechanism: is precisely a dot product — it measures the similarity between the “query” and the “key”, and the higher the similarity the larger the attention weight. This is exactly consistent with the geometric intuition that “the length of a projection = similarity”.
Core Conclusions
- Dot product: multiply corresponding components and add (algebraic)
- Dot product = length of the projection × the original length (geometric)
- Perpendicular ⟺ the dot product is 0
- Duality: a linear transformation from vectors to numbers = taking a dot product with some vector
- The dot product is the cornerstone of similarity in machine learning
| Property | Formula |
|---|---|
| Symmetry | |
| Distributivity | |
| With scalar multiplication | |
| With the norm |
Symbols Used in This Article
| Symbol | Type | Pronunciation/Explanation | Meaning in This Article |
|---|---|---|---|
| Mathematical symbol | v dot w | The dot product: multiply corresponding components and add, giving a scalar | |
| Mathematical symbol | sum v_i w_i | The algebraic expansion of the dot product | |
| Mathematical symbol | norm of v | The length (norm) of the vector | |
| Greek letter | Theta | The angle between the two vectors | |
| Mathematical symbol | cos theta | The cosine of the angle in the geometric formula | |
| Mathematical symbol | v perpendicular w | The two vectors are perpendicular (orthogonal), equivalent to | |
| Mathematical symbol | T from R two to R | A linear transformation from the plane to the number line (a linear functional) | |
| Mathematical symbol | 1 by 2 row vector | The 1×2 row vector matrix of the linear functional |
