Principal Component Analysis

What Is PCA?

Principal component analysis (PCA) is an unsupervised technique for dimensionality reduction.

Suppose our training data for a house has these eight features:

Living area
Number of rooms
Number of bathrooms
Lot size
Garage size
Basement size
Neighborhood score
Distance to city center

These features are related. Living area, rooms, bathrooms, lot size, garage size, and basement size all describe the overall size and quality of the house. Neighborhood score and distance to city center both describe its location.

Instead of eight correlated features, we can use PCA to get one or two meaningful features, called principal components:

Principal component 1 might capture overall house size and quality.
Principal component 2 might capture location.

The Core Intuition

Take some two-dimensional data, with feature 1 on the x-axis and feature 2 on the y-axis. To find the coordinates of a data point, we project it onto each axis.

PCA rotates the axes. The rotated axes are the principal components. To find a data point's new coordinates, we project it onto these new axes.

To reduce dimensions, we keep fewer axes. For two-dimensional data, the only option is to go down to one dimension, so we rotate the axes and keep just one principal component that represents the data well. With three or more dimensions, we can choose to keep one, two, or more components.

So PCA does three things:

It finds new axes, the principal components, which are always orthogonal to each other.
It chooses these axes to capture the maximum variance of the data.
It projects the data onto fewer dimensions.

Why We Need PCA

More features or dimensions can cause several problems:

They are harder to work with, and they can hurt the accuracy of ML models.
Redundant features add no benefit, but they do increase the computation cost.
They can add noise and lead to overfitting.
Data beyond two or three dimensions is difficult to visualize.

PCA simplifies the data while preserving the important information. The variance it captures is maximized, and its new dimensions are uncorrelated.

PCA as Taking a Photo

Imagine taking a photo of a group of people. From one angle, there is lots of overlap and you can't see everyone clearly. From another, you only cover a portion of the group. From the best angle, you capture everyone nicely.

PCA is like taking a photo of the data. We want the angle that captures the most spread, or variance, of the data. Each capture is one principal component, and we can take multiple captures from different angles.

Going from 2D to 1D means taking one capture from the best angle. All the data points are projected onto that rotated axis.

If we compare two possible axes, one might give projected points that are clearly spaced out, while the other gives projected points with lots of overlap. The first projection is far better.

The main question is how to find the principal component, which is the new rotated axis. Once we have it, projecting the data is straightforward.

The Mathematical Foundation

PCA is based on three concepts: the covariance matrix, eigenvalues, and eigenvectors. You don't need heavy math to understand PCA conceptually, but let's walk through the idea.

We need to find a vector $w$ , which is the direction of the principal component. First, we need to understand projection.

Scalar Projection

The scalar projection of a vector $x$ onto $w$ is the length of the shadow of $x$ on $w$ . It is a single real number.

If $\theta$ is the angle between $x$ and $w$ , this length is $\|x\| \cos\theta$ . We also know the dot product formula:

x \cdot w = \|x\| \, \|w\| \cos\theta

Rearranging this gives us the scalar projection:

\|x\| \cos\theta = \frac{x \cdot w}{\|w\|} = \frac{w^T x}{\|w\|}

Depending on the angle between the vectors, the scalar projection can be positive or negative.

Vector Projection

The scalar projection gives us a length, but not a direction. The vector projection of $x$ onto $w$ is an actual vector. It points in the same direction as $w$ , or in the opposite direction if the scalar projection is negative, and its magnitude is the scalar projection.

To get it, we multiply the scalar projection by the direction of $w$ , which is $\frac{w}{\|w\|}$ :

\text{proj}_w(x) = \frac{w^T x}{\|w\|} \cdot \frac{w}{\|w\|} = \frac{w^T x}{\|w\|^2} \, w

Since $w$ only represents a direction, we can assume it has a norm of 1, so $\|w\| = 1$ . The vector projection then simplifies to:

\hat{x} = (w^T x) \, w

Minimizing the Projection Error

For each data point $x$ , we get a projected point $\hat{x}$ . The best direction $w$ is the one that minimizes the error between them:

\|x - \hat{x}\|^2 = \|x - (w^T x) w\|^2

Expanding this gives:

x^T x - 2(w^T x)^2 + (w^T x)^2 \, w^T w

Since $w^T w = \|w\|^2 = 1$ , this simplifies to:

x^T x - (w^T x)^2

We have many data points rather than just one. So let $X$ be a random vector that all the points come from, and minimize the expected error:

E\big[\|X - (w^T X) w\|^2\big] = E[X^T X] - E[(w^T X)^2]

Our unknown is $w$ , and the first term doesn't depend on $w$ , so we can ignore it. To minimize what remains, we have to maximize $E[(w^T X)^2]$ .

The Covariance Matrix

Let's expand this term. Since $w^T X$ is a scalar, we can write its square as $w^T X X^T w$ :

E[(w^T X)^2] = E[w^T X X^T w] = w^T E[X X^T] \, w

Let $C = E[X X^T]$ . Our objective is then:

\max_{w} \; w^T C w \quad \text{where } \|w\| = 1

So what is $C$ ? Assume the data is centered at the mean, so $E[X] = 0$ . The variance of $X$ is then:

E\big[(X - E[X])(X - E[X])^T\big] = E[X X^T]

So $C$ is the covariance matrix. It measures how features vary together, and high covariance means redundancy.

Eigenvalues and Eigenvectors

To maximize $w^T C w$ while keeping $\|w\| = 1$ , we can use a Lagrange multiplier $\lambda$ . This turns the constrained problem into finding the critical points of a single function:

\mathcal{L}(w, \lambda) = w^T C w - \lambda (w^T w - 1)

Since $C$ is symmetric, the gradient with respect to $w$ is $2Cw - 2\lambda w$ . Setting it to zero gives:

C w = \lambda w

If you've studied linear algebra, you'll recognize this equation. It says that $w$ is an eigenvector of the covariance matrix $C$ , and $\lambda$ is its eigenvalue.

So what is the value of our objective at an eigenvector? Substituting $Cw = \lambda w$ and using $w^T w = 1$ :

w^T C w = w^T (\lambda w) = \lambda \, w^T w = \lambda

The objective equals the eigenvalue. So to maximize $w^T C w$ , we pick the eigenvector with the largest eigenvalue.

Picking the Principal Components

The covariance matrix is symmetric and positive semi-definite. This has two useful consequences:

Its eigenvectors are orthogonal to each other.
Its eigenvalues are non-negative.

If $C$ is a $d \times d$ matrix, it has $d$ eigenvalues. We sort them from largest to smallest, with corresponding eigenvectors $e_1, e_2, \dots, e_d$ :

\lambda_1 \geq \lambda_2 \geq \dots \geq \lambda_d

The largest eigenvalues cover the most variance of the data. For one principal component, we take $e_1$ . For two, we also take $e_2$ , and so on.

These eigenvectors are the principal components. Because the eigenvectors are orthogonal, the principal components are too, which matches what we said earlier about rotating the axes.

To sum up:

Eigenvectors give the directions of the principal components.
Eigenvalues give the amount of variance captured, since $\lambda = w^T C w$ is what we maximized.

How PCA Works

Given a dataset, the steps are:

Standardize the data.
Compute the covariance matrix.
Compute the eigenvectors and eigenvalues of the covariance matrix.
Sort the eigenvectors by their eigenvalues.
Pick the top $k$ eigenvectors, which are the principal components.
Project the data onto these principal components.

Advantages

It reduces dimensionality.
It removes noise from the data.
It speeds up training.
It reduces overfitting.
It helps with visualization.

Limitations

It loses interpretability. If we convert 8 or 10 dimensions into 2, we don't know exactly what the new dimensions mean.
It is linear only. It isn't ideal for nonlinear data.
It is sensitive to scaling. We need to scale the data properly before applying PCA.

When to Use PCA

Use PCA when:

The data is high-dimensional.
The features are correlated.
You need to visualize the data.
You need a preprocessing step to go from higher to lower dimensions.

Avoid PCA when:

Interpretability is crucial.
The data is nonlinear.

PCA in Practice

PCA is used for:

Image compression
Face recognition
Noise reduction
Feature preprocessing
Exploratory data analysis

Summary

The key ideas are:

PCA is an unsupervised technique that reduces dimensions by rotating the axes and keeping the most useful ones.
Principal components are orthogonal, and they capture the maximum variance of the data.
Minimizing the projection error is the same as maximizing $w^T C w$ , where $C$ is the covariance matrix.
The solution is $Cw = \lambda w$ , so the principal components are the eigenvectors of $C$ with the largest eigenvalues.
PCA should be used on scaled data, and it isn't ideal for nonlinear data or when interpretability is crucial.