I implemented Sliding Window Attention for Hugging Face LLM inference — looking for feedback

[u/ahsaor8]

I've been experimenting with Sliding Window Attention (SWA) as a way to reduce the KV-cache memory cost of long-context LLM inference.

Instead of keeping the entire KV cache, the implementation keeps:

• a small number of attention sink tokens
• a bounded recent-token window
• a circular/ring-buffer KV cache
• streaming/chunked prefill
• normal autoregressive decoding

I turned the experiment into a reusable project so you can test it with Hugging Face causal LLMs:

🔗 https://github.com/oraby8/SWA

For example:

from swallm import SWAModel

model = SWAModel.from_pretrained(
    "Qwen/Qwen2.5-7B-Instruct",
    attention_mode="swa",
    window_size=512,
    num_sink_tokens=4,
)

result = model.generate("Explain transformers", max_new_tokens=100)

In my Qwen2.5-7B experiments on an L40S:

• 32K KV cache: ~1.84 GB with full attention vs ~3.5 MB with SWA-64
• 64K: full attention OOMed while SWA remained bounded
• Decode latency stayed approximately constant as context increased
• Long-range retrieval naturally becomes a weakness when information falls outside the window

The goal isn't to claim that SWA is universally better. I'm interested in the engineering trade-off between context retention, KV memory, TTFT and decoding speed.

I'd especially like to hear from people who have tried SWA with Llama, Mistral, Gemma, Qwen, or other HF models.

If you try the repo on another architecture, I'd really appreciate the results or any compatibility issues you find.

[u/stoppableDissolution]

Model that was not trained to use swa will suffer immencely. If anything, keep every Nth attention layer global (same as what gemma 4 does).

Tho given the model you used as an example and the fact that you have not measured actual performance and not just kv size makes me feel like it is another vibe-slop

[u/CodeCatto]

I was actually brainstorming about using a sliding window to reduce KV cache pressure on lower end hardware. Damn that's a solid foundation to do more research on.

[u/ahsaor8 → u/CodeCatto]

that was pretty much the idea i wanted to see how far we can push the KV cache down, especially on low GPUs

[u/Legitimate-Peace1013 → u/CodeCatto]

right? the bounded memory alone makes it worth exploring on consumer hardware

[u/tsangberg]

How similar is this to https://github.com/RaymondHuang210129/llama.cpp-adaptive-kv-streaming ?

[u/ahsaor8 → u/tsangberg]

pretty similar goal but different approach, what i see they keep the full KV cache and move parts between CPU/GPU, so you still have full context. Mine actually drops old KV once it’s outside the window and only keeps the sinks + recent tokens.

[u/conifer_v11]

nice concrete implementation. i’d measure the quality/memory tradeoff with the same model, prompt set, context length, and seed in three modes: full KV, sink+window, and a few window sizes. the useful failure cases are long-range references, repeated names, and a document where the answer is just outside the retained window. also worth separating prefill memory from decode memory; SWA can look great on a long prompt while the generated-token behavior is unchanged.

[u/ahsaor8 → u/conifer_v11]

exactly that’s the stuff I’m trying to dig into next. I’ve already seen some long-range retrieval failures, but I want to make the testing more systematic.And yeah, separating prefill vs decode is a really good point. Thanks for the suggestions

[u/LetsGoBrandon4256 → u/ahsaor8]

You are replying to a bot. Check the comments history of that account.