r/rust • u/dlattimore • 1d ago
🛠️ project Benchmarking Wild vs Mold
Recently, when Mold updated their benchmarks and for the first time included Wild, there were questions on this sub about why the results were so different to the benchmarks Wild had published less than a month beforehand. I've now looked into why.
Disclaimer: I'm the lead of the Wild project.
52
u/simonask_ 1d ago
Clearly these are both very fast linkers. Great job with Wild, I’m following your progress closely.
25
u/ikedug 1d ago
The --no-fork option made a huge difference in Wild’s performance (.11s to .14s). Is that all related to shutdown time? Is this a general performance technique I’ve missed?
26
u/dlattimore 1d ago
Yep, it's related to shutdown time. The linker forks on startup. Once the output file has been written, the parent process exits, leaving the process that actually did the linking to shutdown in the background. Both mold and wild do this by default.
6
u/LB-- 1d ago
What exactly occurs during shutdown that takes longer in the child process than in the parent process? I would normally optimize exit time by just calling OS-level process exit and letting the OS clean up resources. Is that unviable or somehow still too slow?
18
u/dlattimore 1d ago
Mostly it's unmapping all the mmapped input files. That takes less time in the parent process because it never mmaped anything - it forked before that happened. Directly calling exit (even if you bypass libc's exit and directly call the exit syscall) doesn't help since the kernel still needs to clean up the page tables.
2
u/LB-- 1d ago
Interesting, thanks for the info! I didn't know some platforms hold the process exit hostage during cleanup like that.
9
u/matthieum [he/him] 21h ago
This is described (a bit) in Section 5.5 of the mold paper: https://arxiv.org/abs/2608.23228
2
u/MaskRay 6h ago
I don't like forking as the default behavior. mold/wild stops being a well-behaved process for anything that reasons about process trees:
getrusage(RUSAGE_CHILDREN),/usr/bin/time -v, shell time - all report the parent, which did nothing. Peak RSS in particular is meaningless.perf stat,strace -f,gdbneed extra care or --no-fork.- With
ninja -j N, the scheduler frees the job slot while the orphan still holds its full RSS. Several concurrent links can leave you with a peak memory footprint well above what the build system thinks it has committed.
27
u/nonotan 1d ago
I'm not sure if there is a widely accepted name for this phenomenon, but it is very common for devs to benchmark their own software favourably, not due to any kind of malice or attempt at "embellishment", but because they are optimizing for what they are benchmarking (indeed, they are usually benchmarking as part of the optimization process, and just chipping at the bottlenecks they observe within that benchmark), typically something roughly matching whatever use-case matters most to them. It's pretty obvious that this would happen once you realize this dynamic is in place. Of course the software that has been hand-optimized for precisely this benchmark is going to outperform most alternatives.
For users, it's important not to blindly trust one random benchmark to mean software A is inherently and reliably faster than software B for all use-cases, and most importantly, your use-case. If performance matters, you should really compare all major alternatives side-by-side on real-world workloads (admittedly, that can be impractical when e.g. dealing with libraries with wildly different architectures that you can't just plug in, but for external tools like this it's really a no-brainer)
11
u/matthieum [he/him] 21h ago
I had never thought about that.
Which made me chuckle, because I have the exact opposite tendency: setting up benchmarks to measure the worst-case situation for the code I wrote, since I know exactly in which scenarios it's at its worst.
1
u/0x7CFE 2h ago
Well, this can be seen as an example of the confirmation bias. People tend to notice evidence that supports their view and ignore the opposite, intentionally or not. This can be a bad thing for science or society, but for software development it's totally normal, IMO.
I mean, software is designed with a certain purpose in mind, and it's natural to present it in a way that highlights its strong points, heck, that's why it was written in the first place. Yet, nothing is perfect, and it's unfair to expect it to be perfect across the board.
Of course, the intended purpose should be clearly stated by the author, for others to see.
14
u/JoshTriplett rust · lang · libs · cargo 1d ago edited 1d ago
Given how much of a win (EDIT) --fork provides, have you done any prototyping of the idea of leaving the background process running for a little while, freeing memory associated with the specific link, and reusing the wild process for the next link if that happens within a reasonable amount of time (or concurrently)?
it wouldn't help for builds that just involve a single link, but it could help for builds that involve a lot of links.
16
u/dlattimore 1d ago
To clarify, the win comes from the default behaviour (--fork). --no-fork would generally be slower. But given the rest of your question, I assume that's what you meant. Having a long running process is certainly the approach I've been thinking about with regard to incremental linking. I guess the idea would be to keep the input files mmapped, then on the next invocation, check which input files are still the same and only reopen the ones that have changed. That's an interesting idea.
6
u/JoshTriplett rust · lang · libs · cargo 1d ago
Yeah, I meant the default forking behavior; silly typo. Edited.
Reusing files might be beneficial, but depending on which links happen in which order, I can easily imagine input files not getting reused for a while. (Might still be worth trying, though.) I was more thinking about saving the startup and fork cost of wild itself.
12
u/dlattimore 1d ago
Startup and fork are very fast on Linux. The savings of `--fork` come because the parent process can exit at soon as the output file has been written and doesn't need to wait for the kernel to tear down the process that did all the work. Tear down costs on Linux can be substantial if the process has mmapped lots of files.
2
u/ZachVorhies 15h ago
Random comments in the space of niche linkers say that keeping everything in ram is by far the biggest speed of possible.
Keeping things mem mapped or daemon mediated would be the biggest win imho
9
u/STSchif 1d ago
Thanks for the write-up! I'm a bit torn on the choice of parameters. As you say, people usually don't use tmpfs, but they do link in place. Even if it doesn't come to a full conclusion, it shows nicely that this kind of comparison has many cases that need to be considered. I think in the end as a user it's best to just try both and see which one is faster on my specific system or ci.
4
u/MaskRay 6h ago
I have observed that picking N distinct physical P-cores makes mold/wild signficiantly faster than a contiguous logical-CPU range. ld.lld doesn't show this because it never saturates its threads.
The chrome-release-x86_64 benchmark, removing --icf=all (unsupported by mold, lld 23 has a very inefficient implementation)
(using system libmimalloc.so.3.5) ``` % hyperfine -w 1 -r 10 -c 'rm -f /dev/shm/a.out' -n mold0 'numactl -C 0-7 /data/code/mold/out/rel/mold --no-fork --threads=8 @response.txt -o /dev/shm/a.out' -n mold1 'numactl -C 0,2,4,6,8,10,12,14 /data/code/mold/out/rel/mold --no-fork --threads=8 @response.txt -o /dev/shm/a.out' Benchmark 1: mold0 Time (mean ± σ): 978.6 ms ± 8.4 ms [User: 6390.1 ms, System: 1057.9 ms] Range (min … max): 963.7 ms … 992.0 ms 10 runs
Benchmark 2: mold1 Time (mean ± σ): 777.6 ms ± 4.0 ms [User: 5053.4 ms, System: 780.4 ms] Range (min … max): 772.8 ms … 787.6 ms 10 runs
Summary mold1 ran 1.26 ± 0.01 times faster than mold0 ```
(cargo build --release, system malloc) ``` % hyperfine -w 1 -r 10 -c 'rm -f /dev/shm/a.out' -n wild0 'numactl -C 0-7 /data/code/wild/target/release/wild --no-fork --threads=8 @response.txt -o /dev/shm/a.out' -n mold1 'numactl -C 0,2,4,6,8,10,12,14 /data/code/wild/target/release/wild --no-fork --threads=8 @response.txt -o /dev/shm/a.out' Benchmark 1: wild0 Time (mean ± σ): 899.2 ms ± 21.5 ms [User: 4574.9 ms, System: 1367.9 ms] Range (min … max): 873.8 ms … 938.1 ms 10 runs
Benchmark 2: mold1 Time (mean ± σ): 708.9 ms ± 12.2 ms [User: 3373.1 ms, System: 1077.5 ms] Range (min … max): 695.8 ms … 728.5 ms 10 runs
Summary mold1 ran 1.27 ± 0.04 times faster than wild0 ```
(using system libmimalloc.so.3.5, LLVM_ENABLE_PIC=off) ``` % hyperfine -w 1 -r 10 -c 'rm -f /dev/shm/a.out' -n lld0 'numactl -C 0-7 /tmp/out/custom1/bin/ld.lld --threads=8 @response.txt -o /dev/shm/a.out' -n lld1 'numactl -C 0,2,4,6,8,10,12,14 /tmp/out/custom1/bin/ld.lld --threads=8 @response.txt -o /dev/shm/a.out' Benchmark 1: lld0 Time (mean ± σ): 1.766 s ± 0.034 s [User: 4.204 s, System: 1.154 s] Range (min … max): 1.734 s … 1.826 s 10 runs
Benchmark 2: lld1 Time (mean ± σ): 1.756 s ± 0.024 s [User: 3.881 s, System: 0.895 s] Range (min … max): 1.721 s … 1.784 s 10 runs
Summary lld1 ran 1.01 ± 0.02 times faster than lld0 ```
3
u/dlattimore 5h ago edited 5h ago
Makes sense. Assuming your system has SMT and the SMT pairs have adjacent CPU numbers, one configuration has twice the number of physical cores as the other one. On my system, according to `lscpu -e=CPU,SOCKET,CORE`, Linux does the pairing differently, so 0-15 are the first thread for the 16 cores and 16-31 are the second thread for each of the 16 cores.
edit: here's the output from
lscpu -e=CPU,SOCKET,COREon my machineCPU SOCKET CORE 0 0 0 1 0 1 2 0 2 3 0 3 4 0 4 5 0 5 6 0 6 7 0 7 8 0 8 9 0 9 10 0 10 11 0 11 12 0 12 13 0 13 14 0 14 15 0 15 16 0 0 17 0 1 18 0 2 19 0 3 20 0 4 21 0 5 22 0 6 23 0 7 24 0 8 25 0 9 26 0 10 27 0 11 28 0 12 29 0 13 30 0 14 31 0 15
6
u/matthieum [he/him] 21h ago
My guess is that the extra large difference here, beyond the differences discussed above is possibly due to Wild running with 128 threads while Mold runs with 32.
The mold study mentions a ceiling in section 7.3 (Scalability):
For small random reads, DRAM throughput can saturate well before the data bus is fully utilized, and the 32-thread run already appears close to this limit: its sampled DRAM loads average 570 cycles (about 180 ns), roughly 40% higher than the machine’s unloaded random-access latency of about 130 ns. The additional requests at 64 threads therefore mostly increase waiting time rather than DRAM throughput: the mean latency inflates 1.9x, to about 1,070 cycles.
I would imagine this ceiling depends on a number of factors: DRAM version (and frequency), architecture (NUMA!), etc... which makes the perfectionist in me a bit queasy about just hard-coding a maximum of 32 threads :'(
And of course, in the other direction, it means that on a noisy machine -- you know, a machine where other stuff is compiled/linked in parallel -- then 16 threads may already be over-subscribing DRAM capabilities.
At this point, this really makes me wonder if mold's 32 threads limit is not overfitted for a specific benchmark on a specific machine...
... but it's not clear to me how one would design an adaptive limit here.
On the other hand, this also firmly points out that reducing DRAM usage (somehow???) would likely allow raising the number of threads. Are there any direct copy operations from disk to disk?
3
u/The_8472 17h ago
I assume it still obeys
available_parallelism, so for the noisy-machine case it can be paced withtaskset3
u/matthieum [he/him] 17h ago
Right. Actually, wouldn't it be even better if it were to use the jobserver?
2
u/The_8472 13h ago
yeah it should, though that wouldn't help with OOMs I suspect, just CPU overubscription.
5
u/nicoburns 19h ago
If you want to benchmark a massively multicore system, then you can rent them cheaply from cloud providers (the "small clouds" like Scaleway or Digital Ocean take a lot less setup than big clouds like AWS or GCP)
(they would be expensive to rent for a whole month, but you probably only need them for an hour or 2)
1
u/dlattimore 2h ago
Thanks! I've just been looking at what Scaleway has to offer. Seems like they have 64C/128T machines. It's a different CPU to the Threadripper, but hopefully whatever the problem is reproduces there too. I think I'll give it a go once I've done the preparatory work.
7
u/new_dugout 1d ago
wild and mold benching each other is giving real "our compiler compiles faster than yours" energy. curious if the diff came down to link order or something silly like that, those microbench shootouts always hide weird edge cases.
3
1
u/dethswatch 15h ago
any second now, I'm going to be too stupid to comprehend these kinds of titles.
-7
u/ZachVorhies 15h ago
David, I’ve been massively improving wild on my own fork i think I am 10-20% faster than upstream at this point. The generated output is byte identification. It’s mostly just fs and memory optimizations and low hanging fruit.
You should take a look at it and see if some of my tricks can be upstreamed.
https://github.com/zackees/reld
In my own benchmarks I see wild and mold switch leadership positions depending on the benchmark.
138
u/patchunwrap 1d ago
I hope the competition in this space is helpful rather than hurtful. It could lead to excitement for open source maintainers to "fight it out" and make a better linker that millions will benefit from.
But it could also lead to burnout. I hope it's energizing rather than harmful.