The Dot Product and Duality

I. The Algebraic Definition of the Dot Product

The dot product of two vectors of the same dimension multiplies the corresponding components and adds them up (the result is a number):

v⋅w=∑i=1nviwi=v1w1+v2w2+⋯+vnwn\mathbf{v} \cdot \mathbf{w} = \sum_{i=1}^{n} v_i w_i = v_1 w_1 + v_2 w_2 + \cdots + v_n w_n
Example: v = (2, 3), w = (4, 1)
v·w = 2×4 + 3×1 = 8 + 3 = 11

Note: the inputs of the dot product are two vectors, and the output is a scalar (a number).

II. The Geometric Meaning of the Dot Product: Projection

The dot product also has a completely equivalent geometric interpretation, best understood through projection:

v⋅w=∥v∥×∥w∥×cos⁡θ\mathbf{v} \cdot \mathbf{w} = \|\mathbf{v}\| \times \|\mathbf{w}\| \times \cos\theta

where θ\theta is the angle between the two vectors.

Intuition: project w\mathbf{w} onto the direction of v\mathbf{v} (or the other way round), and the dot product is “the length of v\mathbf{v} × the length of the projection of w\mathbf{w} in the direction of v\mathbf{v}”.

Angle θcos θSign of v·wMeaning
θ < 90°positivepositiveThe two vectors point roughly the same way
θ = 90°00The two vectors are perpendicular (orthogonal)
θ > 90°negativenegativeThe two vectors point roughly opposite ways

An important conclusion: v⋅w=0  ⟺  v⊥w\mathbf{v} \cdot \mathbf{w} = 0 \iff \mathbf{v} \perp \mathbf{w} (perpendicular ⟺ the dot product is zero).

Verifying perpendicularity:
v = (1, 0), w = (0, 1)
v·w = 1×0 + 0×1 = 0  ✓ the two axes are mutually perpendicular

III. Duality: The Other Side of the Dot Product

Now we can reveal a deep fact: the dot product and a linear transformation are the same thing.

A linear transformation from “vectors to numbers”

Consider a linear transformation T:R2→RT: \mathbb{R}^2 \to \mathbb{R} (the output is a scalar). Its matrix is a 1×2 row vector:

T(xy)=(ab)(xy)=ax+byT\begin{pmatrix} x \\ y \end{pmatrix} = \begin{pmatrix} a & b \end{pmatrix}\begin{pmatrix} x \\ y \end{pmatrix} = ax + by

The key insight: the result of this transformation is exactly the dot product of a vector with (x,y)(x,y):

ax+by⏟T(x,y)=(ab)⏟one vector⋅(xy)\underbrace{ax + by}_{T(x,y)} = \underbrace{\begin{pmatrix} a \\ b \end{pmatrix}}_{\text{one vector}} \cdot \begin{pmatrix} x \\ y \end{pmatrix}

Why this matters for the dot product

This explains why the dot product is symmetric (v⋅w=w⋅vv \cdot w = w \cdot v) and why the direction of the projection can be swapped: projection is essentially a linear transformation, and a linear transformation can be encoded by a vector. The dot product and a linear transformation are two sides of the same coin.

IV. The Significance of the Dot Product in Machine Learning

A measure of similarity: the larger the dot product of two vectors, the more nearly they point the same way (the more similar they are).

ApplicationHow it is used
Cosine similaritycos⁡θ=v⋅w∥v∥∥w∥\cos\theta = \frac{\mathbf{v}\cdot\mathbf{w}}{\|\mathbf{v}\|\|\mathbf{w}\|}, measuring how well the directions agree
Recommender systemsthe dot product of a user vector and an item vector = the degree of match
Neural networkseach output neuron of a linear layer Wx+bW\mathbf{x} + \mathbf{b} is a dot product of the input with a weight vector
The attention mechanismthe dot product of a Query and a Key determines the strength of attention

An example from the attention mechanism: Q⋅KTQ \cdot K^T is precisely a dot product — it measures the similarity between the “query” and the “key”, and the higher the similarity the larger the attention weight. This is exactly consistent with the geometric intuition that “the length of a projection = similarity”.

V. Properties of the Dot Product

PropertyFormula
Symmetryv⋅w=w⋅v\mathbf{v}\cdot\mathbf{w} = \mathbf{w}\cdot\mathbf{v}
Distributivityu⋅(v+w)=u⋅v+u⋅w\mathbf{u}\cdot(\mathbf{v}+\mathbf{w}) = \mathbf{u}\cdot\mathbf{v} + \mathbf{u}\cdot\mathbf{w}
With scalar multiplication(cv)⋅w=c(v⋅w)(c\mathbf{v})\cdot\mathbf{w} = c(\mathbf{v}\cdot\mathbf{w})
With the normv⋅v=∥v∥2\mathbf{v}\cdot\mathbf{v} = \|\mathbf{v}\|^2

Summary

Symbols Used in This Article

SymbolTypePronunciation/ExplanationMeaning in This Article
v⋅w\mathbf{v} \cdot \mathbf{w}Mathematical symbolv dot wThe dot product: multiply corresponding components and add, giving a scalar
∑i=1nviwi\sum_{i=1}^{n} v_i w_iMathematical symbolsum v_i w_iThe algebraic expansion of the dot product
∥v∥\|\mathbf{v}\|Mathematical symbolnorm of vThe length (norm) of the vector v\mathbf{v}
θ\thetaGreek letterThetaThe angle between the two vectors
cos⁡θ\cos\thetaMathematical symbolcos thetaThe cosine of the angle in the geometric formula ∥v∥∥w∥cos⁡θ\|\mathbf{v}\|\|\mathbf{w}\|\cos\theta
v⊥w\mathbf{v} \perp \mathbf{w}Mathematical symbolv perpendicular wThe two vectors are perpendicular (orthogonal), equivalent to v⋅w=0\mathbf{v}\cdot\mathbf{w}=0
T:R2→RT: \mathbb{R}^2 \to \mathbb{R}Mathematical symbolT from R two to RA linear transformation from the plane to the number line (a linear functional)
(ab)\begin{pmatrix} a & b \end{pmatrix}Mathematical symbol1 by 2 row vectorThe 1×2 row vector matrix of the linear functional

Found an issue with this content?

Tell us what could be improved.