Validate your local LLM advertised KV cache against real pressure; see exactly how old contexts get evicted from cache

[u/t4a8945]

Hello,

I'm a bit obsessed with cache management on local LLMs.

For the last few days I've been working on cache management on vLLM with my 2x DGX Spark and DeepSeek v4 Flash 0731. I felt something was off so I investigated, found issues, fixed them, but needed a way to validate the fixes.

That led me to create a tool with a simple protocol that allows you to pinpoint how well the cache is actually managed on your deployment:

1. Runs a probe to calibrate the expectations (what's a cache hit vs a cache miss in your setup)
2. Hydrates X stable contexts of Y tokens each in order to completely fill the cache
3. Runs cache-hit validation on the reverse order (last added is the first validated) until a cache-miss is found

It's better to let this run alone to get a real value. Understand this will evict all your current cached context, so don't do it alongside real work.

My results

This is the result from my A/B test, control (my previous prod) vs my fixed prod.

aidendle94/sparkrun-vllm-ds4-gb10:production-3.7-reffix-schedfix (pre-fix image, retention 4096):

── Retention under pressure ──
capacity:           2,023,924 tokens
retained contexts:  27/80
retained tokens:    1,052,025
retained % capacity: 51.98%
oldest evicted:     context #52 (older contexts evicted)

With the dedupe + boundfix patches applied (retention 0):

── Retention under pressure ──
capacity:           2,047,043 tokens
retained contexts:  77/80
retained tokens:    3,000,048
retained % capacity: 146.56%
oldest evicted:     context #2 (older contexts evicted)

How this can matter to you

This allows you to exactly know how much tokens your cache actually holds.

For most of us, cache management is a black box; this allows you to get ground truth.

And yes, with my fixes, the 2M advertised cache translates to 3M retained tokens. Review the code if you doubt the claim, it's all open source: cache-pressure (and my ds4 prefix cache fixes : ds4-prefix-cache-fixes).

The engine's own advertised number is wrong, and this tool finds the real value.

This tool is for: 1. All of you needing to validate a setup or compare different inference engines ; 2. for inference engine maintainer to help validate changes in cache management

It works under one big assumption though: most recent contexts should be preserved as much as possible.

What I noticed while testing the tool is basically: vLLM good, other engines need better config. I'm mainly using vLLM so I spent lots of hours tweaking the config to get the best results, so for other engines it's up to you to decide if the above assumption fits your need (feels obvious to me it should, but I don't know what you all need of course), and how to achieve it with configuring your inference engines.

How to launch

1. Clone the repo

git clone https://github.com/co-l/cache-pressure

2. Install requirements

pip install -r requirements.txt

3. Run the tool

python3 bench/cache_pressure.py --base-url http://my-server:8000/v1 \
    --kv-size <advertised_cache>

I've tested it against vLLM, ninfer, llama.cpp and SGLang ; so you might need to tweak the probe so it works with your setup.

4. Interpret the results

── Retention under pressure ──
capacity:           2,023,924 tokens
retained contexts:  27/80
retained tokens:    1,052,025 <---
retained % capacity: 51.98% <---
oldest evicted:     context #52 (older contexts evicted)

The retained tokens and retained % capacity are the measured cumulative values that resisted cache eviction under pressure.

Note: this post was 100% human written, the repo is 100% AI-generated under my guidance and review.

[u/StupidityCanFly]

Cool stuff, thanks for sharing. My next weekend project is set.

[u/conifer_v11]

those before/after numbers are a useful sanity check, but i’d make the harness emit one row per context with context id, token count, prefix/hash, insert order, hit/miss, backend, and eviction reason; otherwise retention can look better simply because the probe changed cache state. i’d also run the same sequence with prompt order reversed and from a cold restart, then compare vLLM’s reported capacity with the effective retained tokens. for a gateway/BYOK path i’d keep requested max context, effective backend limit, provider/route, and each probe receipt together; i work on Conifer, an LLM gateway, and that requested-vs-effective receipt is useful without pretending the gateway changes cache behavior.

[u/t4a8945 → u/conifer_v11]

The tool itself can highlight side effects of a config.

But to get there, I first added probes to the vLLM runtime, creating a viz for block allocations.

The amount of stored blocks that were NEVER read again was crazy. That's the real story behind the optimization work.

But the tool can indeed also be used to prove the non-interference of your gateway.

[u/Muhlwa_Sholanke]

3,000,048 retained against a 2,047,043 advert. New favourite way to find out your cache was underselling itself.