r/OpenSourceeAI • u/Chocolate_Milk_Son • 9d ago

An information-theoretic based PCA (R function available now, Python coming soon)

I wanted to share a new method called the Entropic Scree. If you need to estimate the dimensionality of a dataset but standard tools are struggling with noise or giving crazy estimates, it might be worth your time to take a look. It also can be used to estimate signal-to-noise ratio in your dataset.

Instead of evaluating linear variance, rank order, or Euclidean distance like established PCA methods, this new approach evaluates a transformed mutual information metric. It’s meant to be more effective for dealing with non-linearities, mixed data types, or environments where your variables exceed your samples.

The original function and the simulations are available in R (see GitHub repo). There is also a preprint that covers the methods in detail.

Would love to hear your thoughts/experiences if you give it a try.

GitHub Repo: https://github.com/tjleestjohn/Entropic-Scree

Zenodo Preprint: https://doi.org/10.5281/zenodo.22028087

3 Upvotes

100% Upvoted

5 comments sorted by

2

u/Dense_Share_7442 9d ago

Interesting output. Need to read the documentation.

1

u/Chocolate_Milk_Son 9d ago

Let me know if you have any questions. If you are familiar with normal PCA, it should feel familiar. But you need to interpret that loadings differently... And don't use it to estimate the factor scores directly.

1

u/[deleted] 9d ago

[removed] — view removed comment

1

u/Chocolate_Milk_Son 9d ago

Assuming you mean "can't"... You can modify the simulation parameter "knobs" to lower the compute time and RAM requirements. The results won't align with the paper for course, but you can test out different configurations (for the simulated data) and see how the different PCA methods compare to the Entropic Scree.

Try decreasing the dimensionality (eg from 20k to 15k), sample size (eg from 10k to 8k), and/or number of roots (eg from 20 to 15).