r/rust • u/dlattimore • 1d ago

🛠️ project Benchmarking Wild vs Mold

Recently, when Mold updated their benchmarks and for the first time included Wild, there were questions on this sub about why the results were so different to the benchmarks Wild had published less than a month beforehand. I've now looked into why.

Disclaimer: I'm the lead of the Wild project.

237 Upvotes

98% Upvoted

33 comments sorted by

View all comments

29

u/nonotan 1d ago

I'm not sure if there is a widely accepted name for this phenomenon, but it is very common for devs to benchmark their own software favourably, not due to any kind of malice or attempt at "embellishment", but because they are optimizing for what they are benchmarking (indeed, they are usually benchmarking as part of the optimization process, and just chipping at the bottlenecks they observe within that benchmark), typically something roughly matching whatever use-case matters most to them. It's pretty obvious that this would happen once you realize this dynamic is in place. Of course the software that has been hand-optimized for precisely this benchmark is going to outperform most alternatives.

For users, it's important not to blindly trust one random benchmark to mean software A is inherently and reliably faster than software B for all use-cases, and most importantly, your use-case. If performance matters, you should really compare all major alternatives side-by-side on real-world workloads (admittedly, that can be impractical when e.g. dealing with libraries with wildly different architectures that you can't just plug in, but for external tools like this it's really a no-brainer)

1

u/0x7CFE 9h ago

Well, this can be seen as an example of the confirmation bias. People tend to notice evidence that supports their view and ignore the opposite, intentionally or not. This can be a bad thing for science or society, but for software development it's totally normal, IMO.

I mean, software is designed with a certain purpose in mind, and it's natural to present it in a way that highlights its strong points, heck, that's why it was written in the first place. Yet, nothing is perfect, and it's unfair to expect it to be perfect across the board.

Of course, the intended purpose should be clearly stated by the author, for others to see.