Data Science and Engineering

The central task of data science is extracting information from data, and engineering systems handle images, signals and measurement data every day. The projection, the column space, the singular values and the matrix operations you have studied are precisely the basic tools of these tasks.

I. Least Squares: Fitting a Straight Line

Given a number of data points, you want the best-fitting straight line. The error is measured by the length of a vector, and the optimal solution is:

w=(XTX)−1XTy\mathbf{w} = (X^TX)^{-1}X^T\mathbf{y}

II. Image Processing: an Image Is a Matrix

  • an image = a matrix (of pixel values)
  • image compression: SVD keeps the first kk singular values (preserving the main information)
  • convolution = a sliding-window matrix computation

An image is a matrix: a greyscale image is directly a matrix of pixels, and a colour image is three of them (R/G/B). Every image-processing algorithm is, in essence, a matrix computation.

SVD compression: performing a singular value decomposition A=UΣVTA = U\Sigma V^T of the image matrix and keeping only the kk largest singular values (together with their corresponding singular vectors) approximates the original image with far less data — the smaller kk is, the higher the compression ratio and the blurrier the image.

III. Convolution: a Sliding-Window Matrix Computation

The convolution in a convolutional neural network (CNN) slides a small matrix (the kernel) across the image matrix, performing a “multiply the corresponding elements and sum” computation at each position:

yij=∑m,nkmn⋅xi+m,j+ny_{ij} = \sum_{m,n} k_{mn} \cdot x_{i+m, j+n}
  • the kernel kk = a small weight matrix (such as 3×3)
  • each output pixel yijy_{ij} is a weighted sum of the kernel and the local window
  • from the point of view of linear algebra: convolution = a small matrix computation with shared weights

Summary

Symbols Used in This Article

SymbolTypePronunciation/ExplanationMeaning in This Article
y\mathbf{y}Mathematical symbolyThe vector of observed values (the target to be fitted)
XXMathematical symbolXThe design matrix: each row is the features of one sample
w\mathbf{w}Mathematical symbolwThe weight vector of the fit
XTX^TMathematical symbolX transposeThe transpose of the matrix XX
(XTX)−1(X^TX)^{-1}Mathematical symbolX transpose X inverseThe inverse matrix in the normal equations of least squares
U,Σ,VU, \Sigma, VMathematical symbolU Sigma VThe three factors of the singular value decomposition A=UΣVTA = U\Sigma V^T
σi\sigma_iMathematical symbolsigma iThe ii-th singular value, measuring how much information the corresponding direction carries
kkMathematical symbolkThe number of singular values kept when compressing
yijy_{ij}Mathematical symboly i jThe pixel in row ii, column jj of the convolution output
kmnk_{mn}Mathematical symbolk m nThe weight in row mm, column nn of the convolution kernel
xi+m,j+nx_{i+m, j+n}Mathematical symbolxThe pixel of the image corresponding to the kernel’s window

Found an issue with this content?

Tell us what could be improved.