🍃PCA

PCA (Principal Component Analysis) is a popular dimensional reduction method. It outputs several principle components, PC1, PC2, ..., PCn.

Each principal component can be thought of as a line in the Cartesian coordinate system. The objective is to minimize the distances between the data points and their projections onto this line, while maximizing the spread (variance) of the projected data along the line. By doing so, the principal component captures the maximum possible variance in this new dimension. PC1 captures the largest proportion of the data's variance, followed by PC2, which captures the highest proportion of the remaining variance. This pattern continues with each subsequent principal component (PCx). This approach allows PCA to sequentially capture the maximum possible variance in each dimension, thereby minimizing overall information loss.

Dimensional reduction is not only used to project data into lower dimensional plots, but also can be used to reduce the number of features used in other machine learning models, especially in cases when models suffer from the curse of dimensionality. Therefore, dimensional reduction sometimes can improve model performance.

To fit PCA in python, the number of principle components n_components can't be larger than the number of features in the data. There're 16 features in the data. The code below is trying to fit PCA and output 16 principle components.

The generated principle components and their explained variance look as below:

Additionally, if you choose a value x for n_components within its limit (16 in this case), PCA will retain the same variance for all 16 principal components but only output the top x components. For instance, if we set n_components = 3, the resulting 3 principal components will have the same variance as the top 3 components shown in the plot above.

If we plot the top 3 principle components in 3D space, and color the 2 classes in label "deposit", it looks as below:

🌻 Check PCA code here >>

Is there a dimensional reduction method that can better separate the two classes in 3D space, or is it simply not possible to differentiate them effectively in three dimensions? Let’s explore further to find out!