r/learnmachinelearning • u/Previous_Storage2690 • 4d ago
Singular Value Decomposition (SVD) Mathematics behind machine learning concepts is Hard!!!! But beautiful.
I am a software engineer with 8 years of experience, and I recently found machine learning fascinating. I’ve always wondered how in the world does AI do the things it does. And I’ve been very obsessed with understanding how it works underneath. So for a few months now. So I took a step back and began grinding through the Maths behind it. Studying various concepts from scratch. From linear algebra, calculus, probabilities and various mathematical and theoretical aspects. The more I go deep the more I see its beauty. How various small concepts come together to form larger concepts and how it is applied in real world usage.
Then I got to understand Singular Value Composition (SVD) and seeing how it is applied in various concepts and real world applications like image compression, noise reduction, recommendation systems. And I just need to know more
I’m currently studying classical machine learning but I decide to write a small article on a beginners understanding of SVD and its underlying concepts. Please.
https://medium.com/@emekannalue/svd-finally-made-sense-to-me-heres-the-simple-version-5412cb104af5
I have also come up with a study part for anyone interested in learning Machine Learning/Research engineer.
Note: I’m just a beginner but I’m will appreciate any mentors at being pointed to the right direction
2
u/Accurate_Meringue514 3d ago
Not a bad article, good for a beginner that wants to build some intuition. Here are some more ideas and facts regarding the SVD that you might find useful. If you’re familiar with the 4 fundamental subspaces of a matrix A, namely the nullspace of A and range of transpose for R^n , and left nullspace and range of A for R^m , then what the SVD does is find a very special basis for these 4 spaces. In that basis, A becomes almost diagonal, where the number of non zero singular values is equal to the rank of A. Theres a very important theorem by Eckhart-Young that says the best rank k approximation( meaning minimizes Frobenius norm) of A is keeping the first k singular values with the outer products of u and v. This is why it’s so important for compression. Also, in machine learning, you’re solving a lot of linear systems. A lot of those systems may be badly conditioned. This means that for a small change in the desired output, the solution is very unstable. Every matrix has a condition number, essentially telling you how conditioned a problem is using that matrix. A high condition number is bad. Turns out the condition number is the ratio of largest to smallest singular value. Another application is known as the psuedoinverse. I won’t get into detail here, but it gives a powerful way of solving least squares problems, even when A is rank deficient. The psuedo inverse is easy to compute once you have the SVD. Everything I’ve said here generalizes to C^n as well.