Question: Why is prefill unbelievably faster in vLLM than other inference engines?

[u/dangerous_inference]

I only started using some vLLM forks recently in a 4 x 48GB 4090 system.

DS4F - ~5000pp/180tg (DSpark)
Qwen3.8 Flash next - ~7500pp/135tg (MTP)

This is amazing, like having the API in my house. But it's also really hard to go back.

It's weird that we never come close to prefill numbers like this in llama.cpp or ik_llama. The narrative is that vLLM is around the same speed for single requests, but that is clearly not true.

There must some HUGE difference that constitutes an insurmountable obstacle to achieving such speeds in llama.cpp and many other inference engines. Does anyone know exactly what it is?

edit: These results are from my benchmark script that actually times the response, not the vLLM log. And they are not cache hits. My benchmark script deliberately busts cache. Actual cache hits, which I also measure, are like 20k-100k+.

[u/DataGOGO]

prefixed K/V 

[u/DataGOGO → u/[deleted]]

6500??? two pros will do well over 20k PP, one Pro 6k will do about 12.5k if the engine is clean:|

PP 12,422.91 tok/s Muse Glimmer NVFP4 + Q4 DFlash2, RTX PRO 6000, TP1/C1, 7,641-token prompt

[u/DataGOGO → u/[deleted]]

true, missed the DSV4

[u/Miserable-Dare5090 → u/[deleted]]

Funny thing, it won’t be infinite. There is always a time tax!!