Machine Learning and Artificial Intelligence

Machine learning and artificial intelligence are the most “prolific” application area of linear algebra. The matrices, the column space, the eigenvalues and the least squares that you have studied all put in an appearance here.

I. Data Is a Matrix

A dataset is naturally a matrix:

X=(x11x12⋯x1dx21x22⋯x2d⋮⋮⋱⋮xn1xn2⋯xnd)X = \begin{pmatrix} x_{11} & x_{12} & \cdots & x_{1d} \\ x_{21} & x_{22} & \cdots & x_{2d} \\ \vdots & \vdots & \ddots & \vdots \\ x_{n1} & x_{n2} & \cdots & x_{nd} \end{pmatrix}
  • each row = one sample, each column = one feature
  • the row space and the column space correspond directly to the structure of the data

II. A Neural Network = A Chain of Matrix Operations

A layer of a neural network is an affine transformation:

h=σ(Wx+b)\mathbf{h} = \sigma(W\mathbf{x} + \mathbf{b})
  • WW = the weight matrix (a linear transformation)
  • b\mathbf{b} = the bias (a translation)
  • σ\sigma = the activation function (the nonlinearity)

III. The Linear Algebra behind the Classical Models

Model / techniqueThe linear algebra used
Linear regressionthe least squares solution w=(XTX)−1XTy\mathbf{w} = (X^TX)^{-1}X^T\mathbf{y}
Principal component analysis (PCA)the eigendecomposition of the covariance matrix
The singular value decomposition (SVD)decomposing an arbitrary matrix as UΣVTU\Sigma V^T, for dimension reduction / compression
The attention mechanismthe matrix operations of Query / Key / Value
Word2Vec word embeddingsthe distributed vector representation of words
PageRankiterating on the principal eigenvector of the adjacency matrix

Summary

Symbols Used in This Article

SymbolTypePronunciation/ExplanationMeaning in This Article
XXMathematical symbolXThe dataset matrix: nn rows of samples by dd columns of features
xijx_{ij}Mathematical symbolx i jThe jj-th feature of the ii-th sample
nnMathematical symbolnThe number of samples
ddMathematical symboldThe number of features
WWMathematical symbolWThe weight matrix, that is, the linear transformation
b\mathbf{b}Mathematical symbolbThe bias vector, that is, the translation
σ\sigmaMathematical symbolsigmaThe activation function, which introduces the nonlinearity
h\mathbf{h}Mathematical symbolhThe output of one layer of the neural network
XTX^TMathematical symbolX transposeThe transpose of the matrix XX
UΣVTU\Sigma V^TMathematical symbolU Sigma V transposeThe singular value decomposition of the matrix AA
Q,K,VQ, K, VMathematical symbolQ K VThe query, key and value matrices of the attention mechanism

Found an issue with this content?

Tell us what could be improved.