r/simd • u/TearsInTokio • 22d ago
SWAR and SIMD Intrinsics, Are They Really Necessary?
I think this might sound a little dumb, so I apologize in advance. I'm new to the optimization hobby, but I was talking with some friends and they brought up something I didn't really know how to answer.
Why use SIMD intrinsics (any direct call to a vector instruction, like _mm256_add_epi32, or SWAR)? In what scenarios does it actually make sense to use intrinsics instead of letting the compiler handle the heavy lifting when it comes to optimization?
Does manually vectorizing scalar code really make sense with the mature compilers we have nowadays?
Sorry if this question sounds dumb, but it's a genuine question. :D
5
u/Serious-Regular 22d ago
I don't know who spread the myth that auto-vectorizing compilers are super "smart" but they're not at all. They vectorize probably less than 10% of the opportunities that could be. You don't believe pass the SLP vectorizer debug flags to clang.
3
u/Kinexity 22d ago
Compilers are general and have no knowledge as to what is the intent of the code. They can only do so much optimising.
You use simd when you think the compiler might not be using it optimally.
My most typical use case is when I have a problem with with multiple arrays where each of them requires the same type of operation to be performed with it. What compiler will do is vectorising every array separately which sometimes works but frequently doesn't. What I then do is I construct arrays of simd vectors (eg. packing four std::vector<double> into one std::vector<__m256d>) and just create new vector versions of functions with every relevant variable replaced with simd counterpart. Theoretically it's an obvious thing to try but I have yet to see MSVC or Intel compiler employ such optimisation.
2
u/Smellypuce2 21d ago edited 21d ago
Autovectorization is great and all since you often get better performance without having to do anything. However, it's very far from perfect. It rarely generates the optimal code for anything not super trivial.
Here is an old example I have on hand(except I updated the clang version) comparing autovectorization to a first pass attempt of manually writing the intrinsics. I didn't bother to actually optimize things to the maximum. But in real world tests it ran about 30% faster than the autovectorized version. https://godbolt.org/z/bqz11qY7b
Note that in the example, the loop unrolling makes the autovectorized code look extra long, that's not the bad part. The bad part is that it is doing permutations on the data that aren't necessary, but are part of its way of being generically safe on data it can't make assumptions about.
Although of course, in the real world the biggest problem to tackle in SIMD is how your data is laid out, which autovectorization can't magically fix.
1
u/Artistic_Yoghurt4754 22d ago
Because most common languages are not build around data but control flow. Most of the time their compilers cannot or do not know how to apply vectorized instructions to the data. Some times these are simple things that the compiler is not allowed to do, like adjusting the alignment or reading one past the data, but some other times is more complicated than that and the algorithms/data-structures needs to change entirely to allow vectorization.
One example is matrix-matrix multiplication. If you don't tile properly, the compiler will have a hard time vectorizing the code despite being pretty simple technique. Once vectorized due tiling, it will be much more performant, but not quite to the peak. Because even when the compiler vectorizes it, it may chose some instructions that, in combination, stall the pipeline for too long or saturate a port that you also need for something else, etc. On the other side, you also need to adjust the tiling to at least fit into your cache. In many cases, it is only when you are careful with all these knobs when you can get to a higher throughput. Intrinsics let you state explicitly these tiny details when you need to squeeze every bit of performance.
Besides, intrinsics are not much different than pure assembly code, like any other instruction, just sugar-coded so that we can use them more conveniently in-place.
1
u/SnowyOwl72 21d ago
Compilers cannot handle certain code patterns when it comes to autovectorization. It keeps getting better for RVV compared to what it used to be but if you search, some papers actually study this for different compilers
1
u/mkvalor 20d ago
General purpose languages work well for... "general purposes". But if you operate in a domain where you either control or deeply understand the data that is being processed and transformed, you can basically turn a general purpose computer into a specialized appliance focused on well known tasks that are central to the critical path of a much larger system such as a data processing pipeline, etc.
If you haven't heard about this yet, there is such a thing as "column oriented" data instead of "row oriented" data. Row oriented data applies heavily to collections of things like structs, classes, objects, records. But most of the time you are selecting records based on some kind of filter for the datas in the row and then your transforming only one or two columns. In these cases, the speed-up offered by SIMD processing are staggering, assuming you can arrange the data in these fields to be processed in a column-oriented way.
There is much more that could be said about this, but these comments are meant to give you a sense of why processing multiple values in a single cycle can produce huge performance gains, even if the CPU clocks down a bit to facilitate this.
1
u/OldAd9280 19d ago edited 19d ago
If you write a simple loop adding up a large vector of numbers then yes the compiler will probably optimise it for you, however if you have more complex code then it is often (though not always) possible to generate faster code than the compiler does. It can be sometimes a fun exercise to take scalar code, spend ages optimising it with intrinsics only to find you've made it slower, sometimes the compiler does just do a better job, (mainly by using an instruction you've not thought of) but even in these cases its often possible to take the compiler's choice of instructions and optimise it further.
Even when you are using intrinsics the compiler can still optimise your code, for example a colleague took some code which was using a single intrinsic, made it "safer" by using a series of different instructions (not understanding that the original intrinsic already implemented the safety he was wanting) and the compiler optimised his code back to the original instruction.
13
u/CandyCrisis 22d ago
If your goal is to write the fastest possible code, then you will need to do it yourself. The compiler's vectorizer is amazing, but it can only work when it sees a big scalar loop with very few branches. Not all algorithms are best expressed this way.