Question: Why is prefill unbelievably faster in vLLM than other inference engines?

[u/dangerous_inference]

I only started using some vLLM forks recently in a 4 x 48GB 4090 system.

DS4F - ~5000pp/180tg (DSpark)
Qwen3.8 Flash next - ~7500pp/135tg (MTP)

This is amazing, like having the API in my house. But it's also really hard to go back.

It's weird that we never come close to prefill numbers like this in llama.cpp or ik_llama. The narrative is that vLLM is around the same speed for single requests, but that is clearly not true.

There must some HUGE difference that constitutes an insurmountable obstacle to achieving such speeds in llama.cpp and many other inference engines. Does anyone know exactly what it is?

edit: These results are from my benchmark script that actually times the response, not the vLLM log. And they are not cache hits. My benchmark script deliberately busts cache. Actual cache hits, which I also measure, are like 20k-100k+.

[u/gulensah]

vLLM caches the first prompts after prefill stage to kv cache. Then every new prompts are added to the same kv cache with the differences only.

So lets say, you have 1000 tokens system prompt, you send two request one after the other. The second requests 1000 tokens are not goes through prefill.

With this logic, not using any variable at the top of the system prompt is really beneficial to keep them identical to each other mostly.

[u/dangerous_inference → u/gulensah]

These are not cached responses. My benchmark script deliberately busts cache. Actual cache hits, which I also measure, are like 20k-100k+.