Machine Learning and Artificial Intelligence
Machine learning and artificial intelligence are the most “prolific” application area of linear algebra. The matrices, the column space, the eigenvalues and the least squares that you have studied all put in an appearance here.
A dataset is naturally a matrix:
- each row = one sample, each column = one feature
- the row space and the column space correspond directly to the structure of the data
Row Space and Column Space
Recall the column space and the null space from this course: the column space of describes “the information subspace spanned by the features”. A great many machine-learning methods (such as PCA) work precisely around this space.
A layer of a neural network is an affine transformation:
- = the weight matrix (a linear transformation)
- = the bias (a translation)
- = the activation function (the nonlinearity)
Core Conclusion
Deep learning = a composition of multiple layers of matrix transformations and nonlinear activations. The core of the efficient computation of a GPU is large-scale matrix multiplication — a GPU is essentially a machine built for parallel matrix operations.
| Model / technique | The linear algebra used |
|---|---|
| Linear regression | the least squares solution |
| Principal component analysis (PCA) | the eigendecomposition of the covariance matrix |
| The singular value decomposition (SVD) | decomposing an arbitrary matrix as , for dimension reduction / compression |
| The attention mechanism | the matrix operations of Query / Key / Value |
| Word2Vec word embeddings | the distributed vector representation of words |
| PageRank | iterating on the principal eigenvector of the adjacency matrix |
Eigenvalues Are Everywhere
Notice the common point in the table above: PCA uses an eigendecomposition, SVD uses singular values, and PageRank uses the principal eigenvector — they are all asking the same question: “what is the most important direction of this matrix?” This is exactly the geometric meaning of eigenvectors and singular vectors.
Symbols Used in This Article
| Symbol | Type | Pronunciation/Explanation | Meaning in This Article |
|---|---|---|---|
| Mathematical symbol | X | The dataset matrix: rows of samples by columns of features | |
| Mathematical symbol | x i j | The -th feature of the -th sample | |
| Mathematical symbol | n | The number of samples | |
| Mathematical symbol | d | The number of features | |
| Mathematical symbol | W | The weight matrix, that is, the linear transformation | |
| Mathematical symbol | b | The bias vector, that is, the translation | |
| Mathematical symbol | sigma | The activation function, which introduces the nonlinearity | |
| Mathematical symbol | h | The output of one layer of the neural network | |
| Mathematical symbol | X transpose | The transpose of the matrix | |
| Mathematical symbol | U Sigma V transpose | The singular value decomposition of the matrix | |
| Mathematical symbol | Q K V | The query, key and value matrices of the attention mechanism |
