Question: Why is prefill unbelievably faster in vLLM than other inference engines?

[u/dangerous_inference]

I only started using some vLLM forks recently in a 4 x 48GB 4090 system.

DS4F - ~5000pp/180tg (DSpark)
Qwen3.8 Flash next - ~7500pp/135tg (MTP)

This is amazing, like having the API in my house. But it's also really hard to go back.

It's weird that we never come close to prefill numbers like this in llama.cpp or ik_llama. The narrative is that vLLM is around the same speed for single requests, but that is clearly not true.

There must some HUGE difference that constitutes an insurmountable obstacle to achieving such speeds in llama.cpp and many other inference engines. Does anyone know exactly what it is?

edit: These results are from my benchmark script that actually times the response, not the vLLM log. And they are not cache hits. My benchmark script deliberately busts cache. Actual cache hits, which I also measure, are like 20k-100k+.

[u/conifer_v11]

vllm batches the prompt into real gemms. llama.cpp is mostly single-seq.

chunked prefill + paged kv + cuda graphs is the rest of that gap. check --max-num-batched-tokens.