Data Science and Engineering
The central task of data science is extracting information from data, and engineering systems handle images, signals and measurement data every day. The projection, the column space, the singular values and the matrix operations you have studied are precisely the basic tools of these tasks.
Given a number of data points, you want the best-fitting straight line. The error is measured by the length of a vector, and the optimal solution is:
Core Conclusion
The geometric essence: projecting onto the column space of (the projected point = the best approximation) — this is the geometric meaning of least squares. The idea of projection that you met in the section “the column space and the null space” is precisely the mathematical core of data fitting.
- an image = a matrix (of pixel values)
- image compression: SVD keeps the first singular values (preserving the main information)
- convolution = a sliding-window matrix computation
An image is a matrix: a greyscale image is directly a matrix of pixels, and a colour image is three of them (R/G/B). Every image-processing algorithm is, in essence, a matrix computation.
SVD compression: performing a singular value decomposition of the image matrix and keeping only the largest singular values (together with their corresponding singular vectors) approximates the original image with far less data — the smaller is, the higher the compression ratio and the blurrier the image.
Why SVD Compression Works
The singular values measure “how much information this direction carries”. The directions corresponding to the largest singular values contain the main structure of the image (light and dark, outlines), while the small singular values correspond mostly to detail and noise — discarding them, the human eye can hardly notice the difference.
The convolution in a convolutional neural network (CNN) slides a small matrix (the kernel) across the image matrix, performing a “multiply the corresponding elements and sum” computation at each position:
- the kernel = a small weight matrix (such as 3×3)
- each output pixel is a weighted sum of the kernel and the local window
- from the point of view of linear algebra: convolution = a small matrix computation with shared weights
Convolution and Linear Transformations
The way a kernel acts is exactly like a linear transformation: a local weighted sum, sliding window by window. What a CNN learns during training is precisely these kernel matrices — they are responsible for extracting features such as edges and textures.
Symbols Used in This Article
| Symbol | Type | Pronunciation/Explanation | Meaning in This Article |
|---|---|---|---|
| Mathematical symbol | y | The vector of observed values (the target to be fitted) | |
| Mathematical symbol | X | The design matrix: each row is the features of one sample | |
| Mathematical symbol | w | The weight vector of the fit | |
| Mathematical symbol | X transpose | The transpose of the matrix | |
| Mathematical symbol | X transpose X inverse | The inverse matrix in the normal equations of least squares | |
| Mathematical symbol | U Sigma V | The three factors of the singular value decomposition | |
| Mathematical symbol | sigma i | The -th singular value, measuring how much information the corresponding direction carries | |
| Mathematical symbol | k | The number of singular values kept when compressing | |
| Mathematical symbol | y i j | The pixel in row , column of the convolution output | |
| Mathematical symbol | k m n | The weight in row , column of the convolution kernel | |
| Mathematical symbol | x | The pixel of the image corresponding to the kernel’s window |
