PCA Part 1: Eigenvalues of Principal Components and Screeplot Assessment

After normalizing all the indicator variables by taking their Z-scores, a PCA was run to first generate the eigenvalue vs. principal component (PC) matrix. The eigenvalues and each principle component are reported as follows in order of decreasing value:

Moreover, the percent of the covariance explained by each has been calculated and is reported as follows in the same order:

With this matrix, we can start to deduce which principal components explain the most amount of variation in the data and are the most informative. However feature selection of which principal components are meaningful varies depending on the specific criteria used to assess them. Three criteria methods will be shown here for the sake of demonstration, the elbow criteria, the Kaiser criteria, and Horn’s criteria. Ultimately the Horn’s criteria will be the determining criteria for reasons explained below. In any case, a screeplot to chart the eigenvalues vs. their principal components can be incredibly informative.

Here we can readily see two interesting observations, that the first PC seems to capture a majority of the variation in the data, and that the eigenvalues drop off in magnitude dramatically after the first PC.

If we first consider the elbow criteria, it would tell us that only 1 PC is meaningful here, the very first PC. This is because the elbow criteria considers any PCs after a “sharp drop-off“ (say the drop off between PC 1 and 2) to not be meaningful, and any PCs before the drop off to be meaningful.

Moving on to the Kaiser criteria, only PCs with an eigenvalue above 1.0 are to be considered meaningful, while the rest are written off as noise. This can be shown on the screeplot with a Kaiser line, the horizontal purple line at eigenvalue = 1:

This Kaiser criteria shows that only one PC, the same as with the elbow criteria, is meaningful here and exceeds the Kaiser line (principal component 2 is just short of the criteria with an eigenvalue of ~0.99).

Yet, both the Kaiser and elbow criteria tend to be regarded as overly liberal criteria, and in other cases might give too much leniency to principal components. Instead the Horn’s criteria may be the most robust method.

The Horn’s criteria works by simulating a noise distribution of the expected eigenvalues for each PC, and so any PC that exceeds its expected value given by the noise distribution is considered meaningful. So, a noise distribution for this screeplot was generated with 10,000 resamples and then imposed upon the screeplot to see which PCs exceed the noise:

From this one can see that, like before with the other two criteria, only the first PC is meaningful as PC 1 exceeds the noise distribution illustrated by the green splotch (SATA) running across the screeplot.

Thus by the Horn’s criteria, we will consider only the first PC to be meaningful enough to sufficiently capture the information conveyed in the original dataset. Now, since this new principal component will be expressed in an entirely new numeric scale or coordinate space different from the original variables, we will need to interpret the meaning of this principle component and its relation to the original dataset now.

PCA Part 2: Loading Matrix Analysis

PCA generates a second important matrix, the loading matrix. By comparing the eigenvectors for each factor, in each principle component, we can observe to which variables in the original dataset the principle component in question “points towards.“ As in what variables in the original dataset have the highest affinity/correlation with a principle component and thus what variables best describe and contribute to the explanatory power of a PC of interest. Since we found only one meaningful PC, we will only examine its loading matrix and plot it on a bar plot for ease of assessment.

This loading matrix shows us that the eigenvectors on factors 8 and 9 have the highest, positive values. Factors 8 and 9 correspond to the variables “P2- Public Services“ and “P3- Human Rights“ respectively. This tells us that these two variables are fairly representative of the information in this principle component, so we would want to interpret and label this principal component with regards to them.

In spite of this, I would interpret this principle component to mean “Humanitarian Public Goods.“ To elaborate, the commonality between the variables public services and human rights is the fact that they both reflect the availability of some kind of public good in a state.

A public good is some kind of good or service that is non-rival and non-exclusive. The supply of the good is irreducible by consumption and it is impossible to prevent a particular person where the good is provided from enjoying its benefits. An example is national security/military defense. Everyone in a country that has some form of military enjoys the service equally; one person enjoying the protection of the military does not prevent her neighbor from enjoying the same benefit since the “supply“ of military protection has not diminished. Moreover, it is impossible to exclude some citizen from enjoying the service so long as they are in the country. A country will be hard-pressed to say person x is not allowed to be protected by the country’s military while everyone else is, that situation is rather absurd.

For contrast, a private good is the complete opposite, a rival, exclusive good/service. An example is any commodity you could buy in a store. If I buy a soda at the store, that is one soda another person cannot enjoy, and I have thus reduced the current supply of sodas in the store. Moreover, it is exclusionary: if one could not pay for the soda, one would be excluded from enjoying the benefits of consuming it.

Linking this back to the original variables, public services are clearly public goods. Everyone benefits from higher availability and quality of healthcare, education, and public infrastructure. And one’s enjoyment of quality education for example does not prevent her fellow citizens from attaining the same benefits. Plus, it would be quite difficult to make any of them exclusionary, a country would be hard-pressed to say x, y, or z citizens cannot use bridge A under any circumstances (if they could it would no longer really be a public service).

Furthermore, human rights are also a form of public goods in a state. Everyone in the state benefits from a more equitable protection in the eyes of the law, and of respect to their human and civil rights. Meanwhile one cannot reduce the “supply“ of it. Moreover, to make these human rights exclusionary would be rather oxymoronic, the point of them is to include everyone under equal protection by the law.

With this argument as to why the two variables are linked as public goods, I want to elaborate why I described them as “humanitarian.” One should note that things such as military protection also constitutes a form of a public good, and this kind of good is distinctly not humanitarian in nature. We would not want to over attribute factor 1 (C1- Security Apparatus), for example, in the loading matrix as it clearly does not have a high affinity for this principal component. As a result, it would be important to make the distinction with the term humanitarian, to reflect that public goods such as infrastructure or human rights are focused on humanitarian concerns, not security concerns.

With all of this in mind, this PCA seems to suggest that states, based on this dataset, can be best described on an axis of how available/well-developed their humanitarian public goods are. This axis seems to capture the most amount of information in the dataset. Moreover, now we just have a singular, new variable to describe the dataset, having successfully reduced the dimensionality of it. We have gone from 12 variables and transformed them to just one.

Now we can take a look at the original data expressed in terms of this new variable and its new scale.

PCA Part 3: Examining the Transformed Data

From PCA we generate an entirely new set of variables with their own numeric scale different from that of the original dataset. These new values would be meaningless without any subjective interpretation. But we have already done that in the previous step by saying the new variable we have distilled from PCA is quantifying the humanitarian public goods (here on out it will be referred to as HPG) in a state. I took the HPG values and aligned them with their respective states. A snippet of this new dataset, in descending order, is reported as follows:

This small snippet shows us that the new scale for the variable ranges from ~ -5.63 to ~6.64, and that based on the context of the original dataset, states with higher, positive scores in this variable have a higher availability of humanitarian public goods and vice versa for lower, more negative scores. This is the signal amidst the noise of the original dataset.

A next step with having this new variable would be to find a way to characterize or visualize it. Being that we essentially now have a 1-dimensional dataset, a histogram would be the immediately obvious visualization to use in this case, along with its median marked off.

This histogram shows us that the empirical distribution of HPG is not necessarily normally distributed and appears to be lightly right skewed (the mean is reported as ~0, while the median is -0.54). However, we would like to be far more specific in characterizing this dataset since a histogram can only tell us so much. Moreover, it would be nice to see if there is some sort of interval divisions we can make along the dataset to “group“ together certain states, akin to a classifier.

Being that a histogram is a few steps away from essentially being a probability density distribution, we might as well treat the HPG variable as a random variable and estimate its probability density function using KDE.

Kernel Density Estimation

KDE estimates the probability density function of a random variable, and we can use the calculations from this to visualize a probability density distribution. But first we need to determine two parameters before we can run our random variable, HPG, through KDE: the bandwidth size and the type of kernel.

The bandwidth parameter determines the size of the kernel or window that KDE iterates over the dataset. This determines how many surrounding data points around any point x to weigh into the estimate of the function at that point. This parameter is quite important since generally speaking, larger bandwidths result in a smoother distribution, while a smaller bandwidth results in a more granular and rougher distribution. In other words, a larger bandwidth generalizes the probability distribution more since more points are considered at any window, and vice versa for smaller bandwidths.

This is essentially a trade off between bias (underfitting) and variance (overfitting); and like with any other model, we would like to strike a balance between the two. Silverman’s rule provides a rule-of-thumb formula to determine an optimal bandwidth, based on the sample size and the inter-quartile range of the sample. We will use this formula to determine our optimal bandwidth for the sake of simplicity, calculated to be ~0.97.

Meanwhile, the “kernel“ is essentially the “shape” of the window iterated over the random variable’s values. This can range from a gaussian/normal kernel, to a uniform/”top-hat” kernel, to even a triangular or pyramid kernel shape. For this we have gone with a simple gaussian kernel again for the sake of simplicity and as it is the the distribution most compatible with Silverman’s rule.

All said and done, with our KDE parameters picked out, we can now estimate and visualize the probability density distribution of the HPG variable.

Unsurprisingly, the shape of this distribution approximates the shape of the histogram we found before. Moreover, it helps give some evidence that the distribution of HPG amongst countries is not normally distributed. Notice the “fat-tail“ on the right side of the distribution as compared to the thinner tail of the left side. Yet we still do see that extreme, high-magnitude values of HPG are rather unlikely compared to more moderate values.

But we can take this one step further by dividing the distribution into intervals and try to categorize countries along the axis of HPG based on those intervals. A normal classifier algorithm like K nearest neighbors would not work well here since our data is 1-dimensional. Mathematically, we might do this sort of “classification“ instead by simply dividing the distribution function by its local minima, and then treat local maxima as a sort of “centroid“ for each interval. With this particular probability density function, we find one local minima at ~2.38 HPG and thus two intervals with local maximas of ~ -1.12 and ~3.13. Plotting these points and shading the two intervals yields this plot:

This plot illustrates that the probability density distribution can be divided into exactly two intervals based on the local minima which the green point represents. The intervals are colored red and blue, while the blue points represent the local maxima for each interval.

We might consider these two intervals to represent a slightly more definitive method to separate states along the axis of HPG, rather than say splitting them above and below the median or mean value of HPG. The local minima and maxima pay more respect to the underlying probability density function after all.

Furthermore, we could say the states in the blue interval represent states with a relatively high availability of HPG and are more stable, and that states in the red interval have a relatively low availability of HPG and thus poor stability. One could also make the argument that these two intervals are distributed differently too.

Going off of this interpretation, we observe that many states simply are not relatively stable nor have a high availability of HPG, and even then the local maxima in the blue interval shows a high concentration of stable states troublingly close to the threshold between stable and unstable states. Meanwhile, on the low stability interval, most states are relatively unstable but many do not appear to have particularly dire situations with an extremely low HPG value as the local maxima is centered at a moderate HPG value of ~ -1.12. In fact, the probability density tapers off rapidly after this point to the right and stabilizes a bit around the local minima. This could be interpreted to mean that many states are in a transitory state of developing their availability of HPG.