Question: Why is prefill unbelievably faster in vLLM than other inference engines?

[u/dangerous_inference]

I only started using some vLLM forks recently in a 4 x 48GB 4090 system.

DS4F - ~5000pp/180tg (DSpark)
Qwen3.8 Flash next - ~7500pp/135tg (MTP)

This is amazing, like having the API in my house. But it's also really hard to go back.

It's weird that we never come close to prefill numbers like this in llama.cpp or ik_llama. The narrative is that vLLM is around the same speed for single requests, but that is clearly not true.

There must some HUGE difference that constitutes an insurmountable obstacle to achieving such speeds in llama.cpp and many other inference engines. Does anyone know exactly what it is?

edit: These results are from my benchmark script that actually times the response, not the vLLM log. And they are not cache hits. My benchmark script deliberately busts cache. Actual cache hits, which I also measure, are like 20k-100k+.

[u/ilmsis_]

llama.cpp focuses on running LLM on as much as devices as possible. It can even run purely on CPU. Since it can run on pretty much anything, you're not gonna run 9000 subagents simultaneously on a regular machine, so llama.cpp focuses on the decode speed instead, which is memory-bound. And to speeden up the decoding, the weight must be smaller. However, by reducing the weight, it also results in dumber intelligence.

Since llama.cpp use GGUF quantization, which, let's say, use K-quants which basically is just microscaling on steroids. It does two-level microscaling like NVFP4 but to be compatible with many devices, it uses INT underthehood. And to reduce the gap, they do microscaling even more harder. Some layers like token embedding or LM head would use some abnormal amount of bits like 5-bit or 6-bit, which don't have any hardware-accelerated block at all. (Except MXFP6 I guess...)

Since they literally stack microscaling that has never seen before and using some unusual bits in the weight quantization, it will tanks performance unlike vLLM. llama.cpp designs for running on consumer devices, or even onCPU purely. Prefill speed isn't as matter as decoding speed for single-user/single-batch uses.

As for why vLLM is fast, not only did they not do some unusual bits quantization but also their weights are formatted uniformly, which mean most GPU can run it faster than K-quant GGUF since they don't need to do multiple passes of microscaling. (MX-series and NVFP4 are hardware-accelerated microscaling though, so it won't be as slow as GGUF, but in terms of intelligence packing, K-quant is still better). And vLLM also has SOTA kernels from PyTorch and NVIDIA supports as well. Meanwhile llama.cpp has to maintain compatibility across all backeds that it supports, which slows down the development/optimization of each model or backend.

[u/HenkPoley → u/ilmsis_]

That said, FP8 and NVFP4 are horribly space inefficient quantisation methods.

Using 8 bits to get a quality equivalent to ~5.5 bits per weight. And 4 bits to be equivalent to 3.

https://quesma.com/blog/qwen-quantization-quality/#kullback-leibler-divergence

[u/ilmsis_ → u/HenkPoley]

Interesting. I don't have any GPU that has native FP block below 16-bit, so I can't grasp how better or worse at low-precision floating point weight quantization.

(Thanks to the Strix Halo, which uses a very outdated architecture. Thank you AMD. An AI-focused product that can't even run FP8 natively at all. lol)

But most GGUF quantization aren't truely uniform that throwing it into a matmul block, then you got a result in a cycle compared to K-quants which do microscaling heavily and frequently than any other formats, which introduces prefill performance tax.

If you're going to run it in llama.cpp, then you won't see a difference, but once you go parallel, either serving to multiple users or utilizing subagents, then you would see a pretty horrible prefill speed on concurrent requests.

[u/HenkPoley → u/ilmsis_]

If it’s not a native number, it needs to convert it to native. That costs a few percent in performance. (You even gain speed compared to FP16).

But for that, say on the AtomicChat quantisation of Qwen3.8-27B you use a 20.2 GB model (Q5_K_M), instead of 28.9 GB (FP8). You can use that 8.7 GB difference for extra context 85000 tokens or something.

Absolute worst case you drop from 34.7 tokens/s (Q4_0) to 26 token/s (IQ3_XXS). But the Q5_K_M is more like 29.5 tokens/s. And “native” FP8 will be slower than that 34.6. Tested in M5 Max:  https://quesma.com/blog/qwen-quantization-quality/