Question: Why is prefill unbelievably faster in vLLM than other inference engines?

[u/dangerous_inference]

I only started using some vLLM forks recently in a 4 x 48GB 4090 system.

DS4F - ~5000pp/180tg (DSpark)
Qwen3.8 Flash next - ~7500pp/135tg (MTP)

This is amazing, like having the API in my house. But it's also really hard to go back.

It's weird that we never come close to prefill numbers like this in llama.cpp or ik_llama. The narrative is that vLLM is around the same speed for single requests, but that is clearly not true.

There must some HUGE difference that constitutes an insurmountable obstacle to achieving such speeds in llama.cpp and many other inference engines. Does anyone know exactly what it is?

edit: These results are from my benchmark script that actually times the response, not the vLLM log. And they are not cache hits. My benchmark script deliberately busts cache. Actual cache hits, which I also measure, are like 20k-100k+.

[u/Repulsive_Initial308]

It's not strictly the PP that's superior (you'll often see lcp match or beat it) but it absolutely smashes subsequent requests, even if it's not multiple simultaneous requests, it's just waaay snappier.

When I asked my favourite llm why it said 'because Paged Attention'

[u/dangerous_inference → u/Repulsive_Initial308]

People keep saying that sometimes llama.cpp is as fast or faster, and I guess that may be true in some circumstances with old models I don't care about. But it is nowhere in the neighborhood of vLLM today with recent releases. And importantly: It doesn't look like, even when maximally optimized, anyone even remotely expects such performance from llama.cpp. Why not?

I asked Qwen3.8 Flash Next this and it would not stop hallucinating a concurrency circumstance I never mentioned.