PCA reduces the number of dimensions in your data while throwing away as little information as possible. Its insight: the directions in which the data varies most carry the most information, and directions where the data barely spreads carry almost none. PCA finds those high-variance directions, ranks them, and lets you keep only the top few.
How it finds the directions
1Center the data — subtract each feature's mean so the cloud sits at the origin. (Often also scale each feature to unit variance.)
2Compute the covariance matrix, a small d×d table of how the features vary and move together.
3Take its eigenvectors and eigenvalues. Each eigenvector is a direction the covariance matrix only stretches; its eigenvalue is the variance captured along that direction.
4Sort the eigenvectors by eigenvalue, largest first — these are the principal components, PC1, PC2, ….
5Project the data onto the top k components, producing k numbers per point instead of d.
The components are orthogonal and ranked
PC1 is the single direction of greatest variance. PC2 is the direction of greatest *remaining* variance that is at a right angle to PC1, and so on. Because they're orthogonal, each component adds genuinely new information rather than repeating the last.
How many to keep — explained variance
Each eigenvalue divided by the sum of all eigenvalues is the fraction of total variance that component explains. If the eigenvalues are [6, 2, 1, 1] (total 10), then PC1 explains 60% and PC1+PC2 together explain 80%. Keeping just those two cuts the data from 4 dimensions to 2 while preserving 80% of the variance — a big compression for a small loss.
Centering and scaling are not optional
Skip centering and the 'directions of variance' are measured from the wrong origin and PCA misreads the spread. And because PCA chases raw variance, a feature in large units (say, salary in dollars) will dominate one in small units (years) unless you standardize features first.
OperationTimeSpace
Covariance + eigen-decomposition · n samples, d features; d³ for the eigen-decomposition of the d×d covarianceO(n·d² + d³)O(d²)
Project to k dims · multiply centered data by the top-k componentsO(n·d·k)O(n·k)
Check yourself
What does the eigenvalue of a principal component tell you?
