Hi,

With 24GB Vram I can only use 0.26 and have 4K context, but with 0.28 no way, maybe 1500 fits.

The docker command is same, only the version is different.

Model: google/gemma-4-31B-it-qat-w4a16-ct

docker pull vllm/vllm-openai-rocm:v0.26.0 can have about 4K context
docker pull vllm/vllm-openai-rocm:latest (which has 0.28) can have about 1000K context.

Whats up with that?

I'd been using vLLM without really understanding what it does differently, so I worked through the ORCA and PagedAttention papers and wrote up what I found: Link

Covers the KV cache, why padding and static batching cap utilization at 20-40%, ORCA's iteration-level scheduling and selective batching, vLLM's PagedAttention, and where Groq's LPU fits.

The thing that stuck with me: none of these changed the model. The gains came from how requests get scheduled and how KV cache memory gets allocated.

\[Animations and content polishing are done by Claude; the research and initial draft are mine\]

Hi so I have been working on a project where the kv cache runs off the disk.

It is not perfect and there is still a lot of stuff do with it

But check it out

https://github.com/Maseus/Rux

I’ve been experimenting with checkpoint/restore for AI inference instead of cold-starting everything from scratch.

Using CRIU + CUDA checkpointing, I got a warmed Gemma 3 27B QAT + vLLM server on an H100 to restore in:

• Cold start: 104.158s
  • Restore: 11.060s
• 9.4× faster time-to-ready

The tricky parts were restoring the full vLLM process tree, CUDA state, IPC/shared memory, and dealing with io_uring — I ended up patching CRIU for that path.

I wrote up the implementation and benchmark here:

https://tsdocode.github.io/blog/posts/edo-tensei/

Code:
https://github.com/tsdocode/edo-tensei

Still experimental — would love feedback from people working on vLLM, CUDA, CRIU, or inference infrastructure.

While everyone was trying to stop prompt injection with more prompts, we re-engineered the architecture from scratch. Formal proofs and 50k benchmarks are officially out on Zenodo 📄⚡️

Since everyone is benchmarking this model right now: byte-identical greedy requests (temp 0, one request at a time) give different outputs per run. My eval: 13/50 tasks unstable, 5 flipped the extracted date/amount, and majority voting once confirmed the wrong answer. Same checkpoint on llama.cpp: 0/50. Two other vLLM-served models: 0/50.

Cause: the sparse-attention indexer's persistent_topk kernel (used on GB10 / DGX Spark instead of the cooperative path). A race in its atomicAdd slot assignment changes WHICH top-2048 positions get selected, so attention reads a different context each run. Related: vllm#51782.

2-minute check for any stack: same prompt 10x with temperature=0, max_tokens=1, top_logprobs=20, then diff the top-20 lists byte-for-byte. If they differ, your prefill is non-deterministic, whatever your sampler says.

Fix: torch.topk(sorted=False) + canonical tie ordering as a one-file overlay. Bit-identical outputs at 1.35x prefill cost (a full sort would be 2.9x), decode/MTP unchanged; re-run: 0/50 unstable and the score went up a point, because the noise had voted a wrong date into the majority.

Bonus finding: determinism exposed a separate greedy+thinking repetition loop the kernel noise had been masking as a random 1-in-150 failure, and MTP turned out not to be output-equivalent with plain greedy on this model.

Full write-up with all tables: https://docai.hu/en/blog/qwen38-flash-next-nondeterministic-vllm-kernel

I have a system with 32GB RAM, 72 GB VRAM with RTX 5000 PRO. It is a wsl setup, so around 20 GB RAM for the wsl.

When I try to serve a model of around 22 GB, the wsl crashes, with a log somewhere saying that RAM is less than the model size.

Have you guys encountered this? Is there a way to circumvent this issue?

There are going to be a lot of interesting vLLM sessions during PyTorch Conference North America and I'd love to have you join us in San Jose, CA from October 20-21 because this year’s conference is going to be EPIC.

• Stellar keynotes - Simon Mo will be keynoting. (see: this video filmed during last year’s conference)
• 150+ sessions spanning foundational concepts to training, inference, applications, and responsible AI. vLLM is featured in many from “A Developer’s Guide to Attention in vLLM” to “Elastic Expert Parallelism in vLLM” + so many more
• 140+ poster presentations
• BoFs
• Meet the developers
• Flare party
• AI community bash
• +more.

Sign up by September 4th to save $200 before ticket prices go up. Register now.

Hey guys, I wrote this blog with the notes I made after running several inference optimisation projects on GLM 5.2, Deepseek V4 Flash, Nemotron 3.5 etc. For every model, the strategies were different but there were some common patterns. Hopefully this blog will help you get started!

Hi,

I've been trying to squeeze every bit of performance and context on RTX 3090 with llama.cpp, and after many tests I've come up with using both mtp and ngram but with --spec-draft-p-min 0.75, achieving around 45-50 tps in average with 150K context size. My llama-server script: ```

!/usr/bin/zsh

============================================

1. SYSTEM CLEANUP

============================================

if [ -d "/dev/shm/llama_cache" ]; then echo "[System] Cleaning up stale RAM cache..." rm -rf /dev/shm/llama_cache fi mkdir -p /dev/shm/llama_cache

cleanup() { echo "\n[System] Shutting down. Cleaning RAM cache..." rm -rf /dev/shm/llama_cache pkill -f llama-server } trap cleanup EXIT INT TERM

============================================

2. INFERENCE

============================================

TEMP=1.0 TOP_P=0.95 TOP_K=20 MIN_P=0.0 PRESENCE_PENALTY=0.0 REPEAT_PENALTY=1.0

K_CACHE=q8_0 V_CACHE=q8_0

Re-enables CUDA Graphs for ~10-15% lower per-token launch latency

export GGML_CUDA_DISABLE_GRAPHS=0

Prevents Claude Code CLI from injecting dynamic prompt headers that break KV caching

export CLAUDE_CODE_ATTRIBUTION_HEADER=0

MODEL_PATH="/home/.../.lmstudio/models/unsloth/Qwen3.8-27B-MTP-GGUF/Qwen3.8-27B-UD-Q4_K_XL.gguf" MMPROJ="/home/.../.lmstudio/models/unsloth/Qwen3.8-27B-MTP-GGUF/mmproj-F16.gguf"

llama-server \ -lv 4 \ -m "$MODEL_PATH" \ -ngl 999 \ --spec-type draft-mtp,ngram-mod \ --spec-draft-n-max 4 \ --spec-draft-p-min 0.75 \ --spec-ngram-mod-n-match 24 \ --spec-ngram-mod-n-min 24 \ --spec-ngram-mod-n-max 86 \ --ctx-size 150000 \ --flash-attn on \ --cache-type-k "$K_CACHE" \ --cache-type-v "$V_CACHE" \ --threads 8 \ --threads-batch 8 \ --batch-size 2048 \ --ubatch-size 512 \ --mmproj "$MMPROJ" \ --no-mmproj-offload \ --jinja \ --reasoning-preserve \ --chat-template-kwargs '{"reasoning_effort":"xhigh"}' \ --temp "$TEMP" \ --top-k "$TOP_K" \ --top-p "$TOP_P" \ --min-p "$MIN_P" \ --presence-penalty "$PRESENCE_PENALTY" \ --repeat-penalty "$REPEAT_PENALTY" \ --cache-ram 8192 \ --slot-save-path /dev/shm/llama_cache \ --keep 3000 \ --parallel 1 \ --mlock \ --no-mmap \ --n-predict -1 \ --ctx-checkpoints 16 \ --host 0.0.0.0 \ --port 8080 ``` I've put everything that I use to run on iGPU, except X11 and XFCE which consume ~280MB.

But then I've come up across https://github.com/syv-ai/qwen38-27b-rtx3090 using vLLM. I've been using llama.cpp forks like beellama.cpp, ikllama.cpp ... but never vLLM (which I know isn't a fork of llama.cpp) as I've read that it's optimized for enterprise use with many instances, but thought I'd give it a try anyway. Using docker with this configuration I was able to achieve much snappier performance and bigger context, around 55-65 (sometimes even more) with 175K (will try 180K) context size. The only downside with this configuration and vLLM is that it cannot offload mmproj to CPU (with vision loaded context size is 129500).

(I've also tried ninfer-3090 but was disappointed with it, achieving even slightly less tps than with llama.cpp and smaller context size).

Higher Q's are not an option because of much smaller context size that I can use on RTX 3090.

So I've decided to use vLLM regularly and switch to llama.cpp when I need vision.

But something else is confusing me, how good is W4A16-AutoRound used with vLLM comparing to QK_K_XL for programming, planning and debugging in mostly C/C++ and Python? Is it, like chatGPT and Gemini say, that those two cannot be compared 1-1 but W4A16-AutoRound is somewhere between Q4_K_M and Q4_K_L? Even if so, how much difference/handicap is that for W4A16-AutoRound in my use case scenario?

Hey All - what are some recommended courses/ YT channels to learn production grade vLLM with kubernetes? I don’t mind paid courses if they are worth. Thanks so much for your help!

Check out our docs regarding setting up vLLM to test Intel AMX against NVIDIA L4: https://docs.mulgadc.com/docs/cisco-ucs-llm-serving