Principal Component Analysis (PCA)

Full Worked Example: 3D to 2D Dimensionality Reduction

 

Step 1 – Compute the Mean Vector

Given dataset:               

                     X = [[2, 0, 0], 

                            [0, 2, 0],

                            [0, 0, 1],

                            [2, 2, 1]]

The mean vector is computed as:

                                            μ = (1/4) Σ xᵢ = (1, 1, 0.5)

Step 2 – Center the Data

Subtract the mean from each observation:

X̃ = X − μ =
[[ 1, -1, -0.5],
 [-1,  1, -0.5],
 [-1, -1,  0.5],
 [ 1,  1,  0.5]]

Step 3 – Compute the Covariance Matrix

Covariance matrix formula:

Σ = (1/4) X̃ᵀ X̃

After computation:

Σ = [[1, 0, 0],
     [0, 1, 0],
     [0, 0, 0.25]]

Step 4 – Eigenvalue Calculation

Solve the characteristic equation:

det(Σ − λI) = 0

Since Σ is diagonal:

(1 − λ)² (0.25 − λ) = 0

Eigenvalues:

λ₁ = 1,   λ₂ = 1,   λ₃ = 0.25

Step 5 – Compute Eigenvectors

For λ = 1:

z = 0  →  eigenvectors lie in XY-plane

v₁ = (1, 0, 0),   v₂ = (0, 1, 0)

For λ = 0.25:

v₃ = (0, 0, 1)

Step 6 – Select Top 2 Principal Components

Choose eigenvectors corresponding to the largest eigenvalues (λ₁ and λ₂).

W = [[1, 0],
     [0, 1],
     [0, 0]]

Step 7 – Project Data from 3D to 2D

Projection formula:

Z = X̃W

Projected dataset:

Z = [[ 1, -1],
     [-1,  1],
     [-1, -1],
     [ 1,  1]]

Step 8 – Explained Variance

Total variance:

λ₁ + λ₂ + λ₃ = 1 + 1 + 0.25 = 2.25

Variance retained (2D projection):

1 + 1 = 2

Explained variance ratio:

2 / 2.25 = 88.9%