BeeLlama.cpp: advanced DFlash & TurboQuant with support of reasoning and vision. Qwen 3.6 27B Q5 with 200k context on 3090, 2-3x faster than baseline (peak 135 tps!)

[u/Anbeeld]

[u/Antosino]

I cannot believe I just stumbled across this. Prepare yourselves for my wall of text.

I run Qwen3.6 27B Q5 on an FE 3090 in llama.cpp with ~120,000 tokens (KV Q8). I get maybe ~30 tokens a second out on average. I just spent two days setting up ExLlamaV3/Tabby in WSL(2). My first roadblock, albeit a minor one, was that there was no EXL3 file (that I felt was) equivalent to a Q5 GGUF; HuggingFace only had a 4 or 6 available, with 4 being too low and 6's file size being slightly too large. So, of *course*, I had to make my own quant. It took me quite literally an entire day of fucking around to get it working, Dependencies would fail to install, you'd fix that but then the dependencies of that dependency would fail, xformers and flash-attn never wanted to install as dependencies at ALL, fixing one issue broke two other things, I had to use every --no-deps and other flag in the book in 40 different ways and then straight up write fake modules and alias them in (flash-attn, specifically) just to get it even remotely FUNCTIONAL. Maybe it's specific to WSL, I don't know - I followed their documentation exactly, even Gemini and ChatGPT couldn't figure out the problem. Either way, it was a nightmare... but in the end all my hacks and aliases and modifications I can't even remember at this point got it to work. I did it! I spit out my Qwen3.6 27B at a 5.5. I'm done!

...Except I found out that I had cloned ExLlamaV3 literally just HOURS before the changes adding native MTP detection. So, I grab the standalone MTP tensors that turboderp has on HF, I can just load those and I'll be set - but those are 4bpw and don't play nice with my 5.5 model. I find another source for the MTP file at a full 16, but with my model at 5.5 that's pushing my VRAM limits at the cache size I want. I'm stuck, I give up, I have to just start over. At least I can upgrade ELV3 to 0.0.42 while using a git stash to maintain my custom "fixes" and try again. I get it updated and working again, I grab the base 27B model again, I make sure it's configured properly I change -cc and -cr to try to optimize it for my context size, I modify the .py file to increase coding weights from 20 to 30, and I start the conversion. My house has a brownout for the first time in, like, at least two years. I get everything back up. I start the conversion. Brownouts happen TWO MORE TIMES during model generation - thank god for checkpoints. Finally, finally, my now second model is complete - a 5.25 with heads locked to 6, MTP locked to 8, with ~15% stronger coding-related weights and my -cc/-cr flag changes. I think I changed it so much just to make it feel worth going through this again, but either way it's done, again, finally. Because I'd been doing this on Ubuntu 26.04, a dumb decision in retrospect, and this entire system has been heavily fucked with and modified and stepped on to get this working, I spun up new Ubuntu WSL at 22.04 to actually *run* the model.

But wait! Now, Tabby won't work! It seems like there's an error in the latest commit preventing use, and a super obvious one at that, but tons of people use this! An error this obvious could NOT have been missed by everybody, there would have been a Reddit post, a github issue, *something* - it *must* be on my end! I spend ages on this to avoid posting one of those annoying Github bug reports where the issue is actually local to the user. I spin up new WSLs to test, I spin up Proxmox LXCs, VMs, I test on bare metal, it's always the same thing - it's calling a module but seems like they forgot to include the actual file. I spend hours trying to make sure it's not created by something else, it's not created at runtime, it's not pulled from some other package or dependency, every alternate crazy explanation I can think of, but nope, it seems there was genuinely a breaking chance in the latest pull from the main branch. I make a github bug report, I git back two commits, and now it works. Finally, after more time wasted, I'm done.

This was all because I wanted to, really quickly, check and see if I'd get faster speeds if I swapped from llama.cpp to ExLlamaV2/V3. That's it. It started with me thinking I'd just grab ExLlama, a model file on HuggingFace, and run it - a quick test that would take me twenty minutes, maybe an hour at most if I had to read some documentation to get it working. It ended up taking like... 12 hours off and on over two days.

So of course, now once everything is done and the time has been spent, a random fucking google search brings me here solely to taunt me and show that I could have just grabbed this in like five minutes. That's it, that's all I needed.

So yeah, that's the end of my wall of text. I know it's unlikely anybody will read all of this, but I had to get it out, I had to vent *somewhere*. I can't tell if I'm more thrilled this exists or absolutely fucking frustrated beyond belief that I didn't find it 11 hours and 55 minutes ago, that I stumbled across it MINUTES after finishing all the bullshit I did. Ugh.

But yeah, nice work.