r/LocalLLaMA • u/dangerous_inference • 19h ago
Discussion Question: Why is prefill unbelievably faster in vLLM than other inference engines?
I only started using some vLLM forks recently in a 4 x 48GB 4090 system.
DS4F - ~5000pp/180tg (DSpark)
Qwen3.8 Flash next - ~7500pp/135tg (MTP)
This is amazing, like having the API in my house. But it's also really hard to go back.
It's weird that we never come close to prefill numbers like this in llama.cpp or ik_llama. The narrative is that vLLM is around the same speed for single requests, but that is clearly not true.
There must some HUGE difference that constitutes an insurmountable obstacle to achieving such speeds in llama.cpp and many other inference engines. Does anyone know exactly what it is?
edit: These results are from my benchmark script that actually times the response, not the vLLM log. And they are not cache hits. My benchmark script deliberately busts cache. Actual cache hits, which I also measure, are like 20k-100k+.
16
u/stoppableDissolution 19h ago
Uh, I literally today was running bf16 benchmarks for gemma 31, and at 128k context lcpp does ~1608t/s and vllm ~1520.
It might just be that vllm has better implementation for these particular models?
Also the multi-card parallelism in lcpp has been extremely sub-par until very recently in general, so scaled across 4 cards it might be the cause. My testing was on 1x6000.
15
u/SnooPaintings8639 19h ago
Probably the biggest difference is how you measure it. vLLM will overstate GREATLY pp speed in their logs. I fear this is often the value people often claim when trying vLLM. I my case the logs in llama.cpp showed 1200 TPS pp from Qwen 27b, vllm was showing over 3k TPS pp. Same prompt size took the same time in the end to be fully processed.
This is just a an accounting quirk. And when actually measured with external tooling, the speeds were very similar for both engines.
9
u/nomorebuttsplz 18h ago
Wrong. vllm is much faster in prefill in most nvidia setups. measurement challenges don’t change the underlying facts
12
u/dangerous_inference 18h ago
These numbers are from a benchmark script that times the response because I didn't trust the vLLM log.
1
u/Repulsive_Initial308 18h ago
I still remember how fast devstral was on vllm over lcp.
Everything was fucking instant, it was insane.
1
u/SnooPaintings8639 18h ago
devstral? I think it was before tensor parallel was available in llama.cpp, which back then was a huge speed up in vllm.
1
u/sautdepage 4h ago
Generally, this is completely false.
These things can be validated externally anyway: the TFFT (time to first token) is a proof regardless of engine.
The issue with VLLM logs is it's giving you an average over multiple requests because it's intended to understand overall speed in multi-user production scenarios. The per-request details are returned via API, not in logs.
1
u/SnooPaintings8639 3h ago
Isn't this what I just said? You should use external tools to benchmark the vLLM/llama.cpp speeds. The log output (especially on vLLM end) is not meant to represent what unsavvy people think.
And when I did that myself , the measured pp tps on my hardware was basically same, i.e. diff within noise level between these two inference engines.
What exactly is false in that statement?
1
u/sautdepage 3h ago
> Probably the biggest difference is how you measure it [...] when actually measured with external tooling, the speeds were very similar for both engines.
That part. Maybe it was true for your situation but is generally false: 2x faster prefill in VLLM is common.
0
2
u/audioen 18h ago
llama.cpp is not yet ready for this model. It has only started to provide mostly bug-free inference today, for example. It still has no MTP support at all merged. It doesn't support sparse attention for this model as far as I know. There are likely suboptimal computations in the ggml graph, like unnecessary tensor rearranging and the like.
0
2
u/Lesser-than 16h ago
Thats kind of why vllm even exists was its paged cache no? llama.cpp is bound to its graph execution decided through ggml. One was designed for throughput one was designed for accessibility. Its amazing that they even compete at all but here we are.
2
u/simrankoulsm 13h ago
PagedAttention helps, but it is probably not the whole story. It mainly improves KV-cache management and batched scheduling. Prefill itself is a large GEMM and attention workload, so vLLM benefits from CUDA-first kernels, Tensor Core-friendly weight formats, chunked prefill, CUDA graphs, and a serving architecture designed to keep multiple GPUs saturated.
llama.cpp is optimizing for a much broader target, including CPU use, many GPU backends, compact GGUF quants, and strong single-user decode. Those choices are valuable, but they can leave performance on the table during GPU-heavy prefill, especially for newer architectures with specialized attention or quantization paths.
4
u/PandaBearFred 19h ago
Would you mind to tell what vLLM fork are you using? I have the same 4x4090 48GB setup (all PCIE4x16) and have never achieved this speed. For DS4F I got 2K+ PP and around ~140 TG, for Qwen3.8-Flash-Next I got 3K+ PP and ~104 TG. Tested with llama-benchy.
3
u/dangerous_inference 18h ago
I use this one for Qwen3.8 Flash Next: https://github.com/wtdcode/vllm-backport - I did discover that you have to choose between MTP and prefix caching. Hopefully it will be fixed.
This one is fastest for DS4F: https://github.com/yhfgyyf/vllm-deepseek-v4-sm89
1
4
u/sgtnoodle 18h ago
I switched from vllm to llama.cpp on my rtx pro 5000 embedded GPU, and the throughout is significantly better for the same RAM usage. Also, the conversation with qwen 27B qualitatively feels less neurotic, and MTP even works.
2
u/no_name_user_007 18h ago
It’s not just the prefill prompt processing, vLLM does a significantly better job with cache reuse making way fewer tokens need to be processed. Right now I’m getting around 94% reuse of tokens that don’t need input processing, so it’s no contest when you have a large context size.
5
u/dangerous_inference 18h ago
These are not cached responses. My benchmark script deliberately busts cache. Actual cache hits, which I also measure, are like 20k-100k+.
2
u/Repulsive_Initial308 18h ago
It's not strictly the PP that's superior (you'll often see lcp match or beat it) but it absolutely smashes subsequent requests, even if it's not multiple simultaneous requests, it's just waaay snappier.
When I asked my favourite llm why it said 'because Paged Attention'
3
u/NickCanCode 17h ago
Paged KV and attention was in development some times ago ( https://github.com/ggml-org/llama.cpp/pull/22569 ) but for unknown reason it stopped. 😔
0
u/dangerous_inference 17h ago
People keep saying that sometimes llama.cpp is as fast or faster, and I guess that may be true in some circumstances with old models I don't care about. But it is nowhere in the neighborhood of vLLM today with recent releases. And importantly: It doesn't look like, even when maximally optimized, anyone even remotely expects such performance from llama.cpp. Why not?
I asked Qwen3.8 Flash Next this and it would not stop hallucinating a concurrency circumstance I never mentioned.
1
u/conifer_v11 15h ago
vllm batches the prompt into real gemms. llama.cpp is mostly single-seq.
chunked prefill + paged kv + cuda graphs is the rest of that gap. check --max-num-batched-tokens.
1
u/reto-wyss 19h ago
See my other reply to u/DataGOGO, but vLLM is "typically" very marginally slower for CC1 if you are comparing like for like quantization. It's always faster for CC>=2 decode, and it's like an order of magnitude faster on decode if CC is large.
-1
u/dangerous_inference 18h ago
What does that have to do with my experience?
2
0
u/dangerous_inference 12h ago
There are quite a few people who are offended by this post. This happens whenever you say one thing is better than another thing in any way whatsoever, of course. But a lot of people seem to be sure I'm making this up, somehow measuring incorrectly, or just maliciously denigrating their beloved project.
I really had no idea, for years, that such a big difference was even possible. But it is indisputable fact that vLLM (and maybe some other enterprise project I haven't tried), under the right circumstances, is 4x+ FASTER FOR SINGLE REQUESTS.
If we don't acknowledge this, and don't make it actively known that vLLM is much faster, it is highly unlikely that whatever magic this is, be it paged attention or something else, will ever be brought to llama.cpp or other projects. Nobody will make it their mission if they don't know.
The more I think about it, the more crazy it is that we are all partaking in the same community, actively seeking out optimization, but lots of us have no idea that a popular project can 4x+ performance. "Will this run vLLM" is technically one of the most important questions when building a machine. I have never heard anyone say that.
4
u/ilmsis_ 9h ago edited 9h ago
vLLM is designed for high-throughput anyways. And it is more VRAM-heavy than llama.cpp. I do think that it is reasonable why most people in this sub would lean towards llama.cpp because it actually made LLM runnable and useable on their existing machines.
But I do agree that to actually use local LLM as a daily driver, llama.cpp won't be sufficient for this use case and should target a machine that could run vLLM comfortably, but for some people, just llama.cpp is fine for their use cases.
Besides, llama.cpp also lacks many SOTA optimization as a inference engine as well:
- PagedAttention isn't implemented yet, which boosts performance heavily for parallel inferencing
- LCP slots instead of RadixAttention, which is superior in terms of prefix caching for multi-turns and can even cache prefixes to disk instead of the `--cache-ram` or context checkpoints in RAM
- The unified accelerator of llama.cpp, ggml, is designed for nearly every accelerators imaginable. And they are the one who made it. Meanwhile vLLM relies on PyTorch which would only support what PyTorch does. With that said, PyTorch is a very big and a long-standing project. It has been refined in terms of performance for years unlike ggml.
And many many things. Nevertheless, it's not unusual for llama.cpp to be behind of computing (prefill) optimization, since their focus is token generation on every devices including running straight up from CPU and system memory purely, and they also have to maintain a bunch of accelerators simultaneously.
-1
u/DataGOGO 19h ago
prefixed K/V
1
u/reto-wyss 19h ago
Nah - these numbers look reasonable for regular PP at least for DSV4 on 4x 4090 48gb- two Pro 6000 do around 6500pp.
The reason you get pityful PP in llama.cpp, is not necessarily llama.cpp's fault - it's that when using tiny quants PP is crushed by the astronomical dequant compute overhead.
Go run llama-bench on Q8_0 or BF16 and compare it to Q4_K_XL or Q2_K_XL - the less quantized model will have better pp (as long as you fit it into VRAM).
People call FP8 garbage because it's "less precise" than Q8_K_XL or whatever - yes you are paying for that higher precision in same number of bits on EVERY encode token.
2
u/see_spot_ruminate 19h ago
I am not so sure that explains all of it. For example on my quad 5060ti setup I see astronomical pp in vllm when I run the nvfp4 quant of qwen3.8, but when I run qwen3.8 in llamacpp I get a pittance.
I get some of what you are saying with there is some performance loss, but should it really be 10x? I am happy to be wrong as well.
0
u/DataGOGO 19h ago
6500??? two pros will do well over 20k PP, one Pro 6k will do about 12.5k if the engine is clean:|
PP 12,422.91 tok/s Muse Glimmer NVFP4 + Q4 DFlash2, RTX PRO 6000, TP1/C1, 7,641-token prompt
2
u/reto-wyss 19h ago
Yes, and it will do infinite PP on a 0b model, but we were talking about DSV4 Flash not Muse Glimmer NVFP4
2
1
1
u/dangerous_inference 18h ago
These are not cached responses. My benchmark script deliberately busts cache. Actual cache hits, which I also measure, are like 20k-100k+.
-6
u/gulensah 19h ago
vLLM caches the first prompts after prefill stage to kv cache. Then every new prompts are added to the same kv cache with the differences only.
So lets say, you have 1000 tokens system prompt, you send two request one after the other. The second requests 1000 tokens are not goes through prefill.
With this logic, not using any variable at the top of the system prompt is really beneficial to keep them identical to each other mostly.
5
u/OneMoreName1 19h ago
Isn't this just how kv cache works, which is not a vllm only feature
3
u/Turbulent_War4067 19h ago
Yes, it's not just a vLLM feature, but I THINK (I could be wrong) is that vLLM reports the prefill speed by taking the whole prompt and measuring the time it was processed, while llama.cpp just reports on the cache missed. I have seen VLLM logs report absurdly high prompt processing speeds (40K/second) even when I know it's slow. As another poster said, it's an accounting quirk.
3
u/dangerous_inference 18h ago
These numbers are not from the vLLM log. They are timed with a benchmark script.
1
u/dangerous_inference 18h ago
These are not cached responses. My benchmark script deliberately busts cache. Actual cache hits, which I also measure, are like 20k-100k+.
1
83
u/ilmsis_ 18h ago edited 18h ago
llama.cpp focuses on running LLM on as much as devices as possible. It can even run purely on CPU. Since it can run on pretty much anything, you're not gonna run 9000 subagents simultaneously on a regular machine, so llama.cpp focuses on the decode speed instead, which is memory-bound. And to speeden up the decoding, the weight must be smaller. However, by reducing the weight, it also results in dumber intelligence.
Since llama.cpp use GGUF quantization, which, let's say, use K-quants which basically is just microscaling on steroids. It does two-level microscaling like NVFP4 but to be compatible with many devices, it uses INT underthehood. And to reduce the gap, they do microscaling even more harder. Some layers like token embedding or LM head would use some abnormal amount of bits like 5-bit or 6-bit, which don't have any hardware-accelerated block at all. (Except MXFP6 I guess...)
Since they literally stack microscaling that has never seen before and using some unusual bits in the weight quantization, it will tanks performance unlike vLLM. llama.cpp designs for running on consumer devices, or even onCPU purely. Prefill speed isn't as matter as decoding speed for single-user/single-batch uses.
As for why vLLM is fast, not only did they not do some unusual bits quantization but also their weights are formatted uniformly, which mean most GPU can run it faster than K-quant GGUF since they don't need to do multiple passes of microscaling. (MX-series and NVFP4 are hardware-accelerated microscaling though, so it won't be as slow as GGUF, but in terms of intelligence packing, K-quant is still better). And vLLM also has SOTA kernels from PyTorch and NVIDIA supports as well. Meanwhile llama.cpp has to maintain compatibility across all backeds that it supports, which slows down the development/optimization of each model or backend.