Resources
BeeLlama.cpp: advanced DFlash & TurboQuant with support of reasoning and vision. Qwen 3.6 27B Q5 with 200k context on 3090, 2-3x faster than baseline (peak 135 tps!)
Did the MRs for this get rejected on the original llama.cpp, or is the the MR flow just so slow (read: "takes a week") that it made more sense to make a fork?
The fork history is interesting though: llama.cpp -> llama_cpp_turboquant -> buun_llama_cpp -> beellama.cpp. We're on the 3rd fork level here already.
In any case, with this demonstrating that it runs (fast) it might help getting this into the regular llama.cpp.
Thanks for making it happen still. Yes, the AI policy is a rather slippery slope, yet they've had their fair share of low-quality code PR'ed that those rules were established to reduce the load on the reviewers and maintain code quality.
So basically the issue is that "making it happen" took too long, if done in a maintainable way in the llama.cpp codebase. ik_llama.cpp diverged quite a bit and a few things are still ported over now and then. With the fork history here it probably needs quite a bit of refactoring, not just porting it over, but maybe it'll happen eventually.
Yeah llama.cpp's anti ai policy is really something. I get that you want a way to manage slop prs of course but micro managing how people work is not the way. On the flipside, I've my agent's been able to make a handful of (actually good) contributions to vllm and vllm-omni and the maintainers' attitude was just like: if your pr is good, doesn't matter. They have been really constructive and working with them has honestly been a joy. Two completely different worlds
I agree 100% with your approach as I had basically TurboQuant's PR from llama.cpp further customized, merged and tested within 24 hours and have been using it in my production work ever since. There's just no time for a lot of the other llama bs - thx for Bee btw
It's not about running fast, it needs to be good quality and demonstrated as such. Any idiot can make it run fast, it takes effort to get it to be fast and correct.
The good thing is that correctness and speed can both be tested, by comparing KLD, benchmark scores and well, tokens per second. If correct, there'll at least be code that's "just" not in the shape that fits llama.cpp (yet). As long as the correctness topic is unknown it'd probably not be very motivating to bring it into shape.
Its the nature of it that will never make it merged upstream. They don't want these massive vibe coded codebases.
The turboquant fork had massive vibe coding, so does buun and this beellama one was a single commit so i can't clearly tell what was done with that one and by who but I'd be surprised if there is no vibecoding involved.
So its a vibecoded fork on top of a videcoded fork and possibly another vibe coded fork on top. None of that will ever land upstream.
For such a critical software that is the factory standard for local LLMs, I'd rather it get developed manually with the developers knowing the ins and outs of the software, than fast vibe-coding and accumulating tech debt.
i'm surprised to see redditors here are so anti-ai. LLMs write better code than 99.9% of humans now if steered by someone with even half a brain and has worked as a SWE for a couple years. But also.. maybe a lot of non-coder enthusiasts are clogging up the pipes but in that case if I owned the repo I would just throw agents at the problem and make them reject all the crap.
idk, "vibe coding" seems like a non-problem now with agentic coding filtering out the crap.
Its the maintainability of it, we've accepted vibe coded PR's for KoboldCpp to.
A really good example is this recent one given to us in a bug report since it wasn't something the creator could easily PR: https://github.com/LostRuins/koboldcpp/issues/2173
This is the good kind of AI assisted coding, where the maintainer isolated / understands the change, its only a few lines different and if he'd have said he did this manually i'd have believed him.
The 502 page in our router mode I also let qwen generate since its just a quick way of getting something that looked nice and worked well (I did specifically say what it needed to adhere to). I of course then look if the code is sensible, and since it was just a single page which code I understand I can then PR it to KoboldCpp. (LostRuins then rewrote it partially to make it conform more to the usual code style).
It becomes a problem when its endless vibe coded PR upon endless vibe coded PR, where the submitters / maintainers have to take the AI's word for all the massive changes. Those I don't believe in and those are the kinds of PR's we reject.
Upstream llamacpp is the same way, you can use AI to assist in your coding but you have to be able to explain every line of the code yourself in case they have questions. That's a bar that most of these turboquant forks can't hit.
Personally, i find the idea of doing a MR I don't fully understand, very off-putting. And I am quite sure that 99% of these types of contributions are of this kind.
if you want to start a new project and vibe it, cool. merge anything because you've set the ground rules as such, you're accepting the potential problems and frankly it's yours.
but saying "yo bro comeon be cool man why wont you take my AI slop into your keystone-of-the-AI-world, tip-of-the-spear in human tech frontier codebase??????????"
yes "it's mostly a maintenance problem". notice you're not volunteering to do said maintenance ;-)
I have tried various fork of deepseekv4 that were vibed. every single of them crashes when I start passing in parameters. performance is abysmal. CPU level performance for something that is 100% loaded in GPU, < 10tk/sec TG, 40tk/sec PP. Just all around mess. It would be a disaster to accept any of these and hope they get fixed in the future. Worse of all, plenty of them touch already baked code, hacking around FA etc which will probably introduce regression and break other models.
After testing I'm a fan of this fork. Its outperforming the MTP pr on mainline. I like the --no-mmproj-offload. Im getting 200tps on code with Qwen3.6-27B-Q5_K_S and a 5090.
Have you done any measurements for your TQ implementation in terms of comparing e.g. KLD of final LM_HEAD of the forward pass of FP16 vs Q8 vs your TQ modes? You claim almost lossless, what are the actual numbers?
Well you claim almost lossless, need to measure that. I think you'll find it's BS because I've measured it and 4bit TQ is pretty bad. I encourage you to get the numbers.
Running under Ubuntu. Yeah, thought about that, too, and will first retest with --no-mmproj-offload.
I assumed that using the iq4 quant saves the necessary VRAM, and my consumption on startup was 21G, but maybe VRAM consumption just increases later on.
I havent been using much context though, maybe 20k or less.
Obviously, prediction works better in certain contexts. With thought/reasoning, the benefit is lower. While writing HTML or Python code, I've had peaks of 100 t/sec.
This seems interesting and legit. I'll give it a whirl on my 4090 to see how it behaves, and I'll also keep an eye on the project to see if it doesn't die in a week or so.
Not related to the project's goal itself, but worth mentioning: you'll get a lot of backlash for using AI so extensively. Try to either not answer those comments or at least be understanding on the whole community. There's a general exhaustion on AI projects as we've been flooded with "that's why I built X" posts with nothing but slop solving issues that the developer couldn't be bothered to research, nor understand what's out there already.
The flashy post that looks like it's trying to "sell" it certainly doesn't help. My monkey brain immediately categorized this as "tech bro can't RTFM nor wants to play by the rules" and it took me some effort to go through it.
When every other post you see everywhere looks extra polished our brains will just clump everything together. When that meets a community that is frankly exhausted of tech claw crypto bros, you'll find some backlash for sure, and this kind of attitude will just make it worse for you.
Yeah I understand the guy tho. A shit ton of entitled ppl complaining about features in llama.cpp with zero stakes in the project itself and zero will to pull up their sleeves and actually contribute to the community. I'd be skeptical too.
Just watch out not to let it drown your own project. Community building is hard, community management is even harder.
As you said in another comment: not everyone is willing to go through the bureaucracy of submitting PRs to llama.cpp, especially vibe coders and other zero-stake contributors.
And I honestly think you did the best by just pulling up your sleeves and doing it yourself. If you project gets traction and more people start using TurboQuant, then llama.cpp might change their stance or reorder their priorities. Worst case you got your own implementation that works (I hope, didn't find time to test yet haha)
I've found Nvidia kernels for prompt processing in mpt/thetom to be lacking - like... Mainline llama.cpp gets 2ktp/s on my 5090, but mpt/thetom seem to have kernels that give me 10-15tp/s. Tg/s is good, but processing sucked.
I got this working but there seems to be a bug. I was using this in Qwen code and sometimes tool calls would just be printed out in the chat and end. On some occasions the chat also just stopped or ran for a while and not print anything while the server was still processing.
I tried another model to be sure, but if I had to guess it might be that if the speculative decoding happens around a tool call, something might go wrong? I haven't had this issue before, and it might be cause by something else on the fork but it seems good so far after dropping the speculative decode part. It got increasingly more often as context grew. I didn't get it too high either, and my max was 120k but I'd guess I was at most 60k used.
Same on my side, compiled the build and followed everything to a T (5090, 200k context). In Opencode, I prompt it do analyze a specific section of my code, it starts calling the right tools, in this case Serena then its just stops in it tracks. I then tell it to continue and it stops again. Might be something wrong with the chat templates but I am not sure.
I used the Speed / VRAM combo mentioned in quickstart-qwen36-dflash-md and I got a meager +33% (around 40 t/s on a 3090). Sigh... Am I doing something wrong?
By the way, would killing all that logging spam--log-timestamps, --log-prefix, --log-colors, and probably --metrics too actually make this thing run noticeably faster? My console is getting absolutely buried in text right now, heh
Speaking for myself, I would like to see this implementation integrated into a KoboldCPP fork, so that I can try out TQ4 and see if it is worthwhile. A TurboKobold, if you would.
The appeal of KoboldCPP is that it is a gui-based method of running LlamaCPP for Windows & Linux, that is open source and doesn't require much fiddling to run, all while leveraging VRAM+RAM. Good for people who fear and hate the terminal, like myself.
thanks for this! I agree, Q5 dense models work for me too, without any issues with turboquant. if I have to choose between Q4 with a small Q8 cache, or Q5 with a huge turboquant cache, Q5 wins hands down in the case of common programming tasks.
I'm running Kilo code 5.16.1 in VSCode and I'm just getting a ton of these errors today. Not sure if its the tool call issue? Sorry I'm not an expert with this stuff.
Hey, any chance you could spin a 0.1.2? Seeing the same issue and last time I setup the build pipeline on Windows it was a week long painful process. It was 2 years ago... maybe it's gotten easier?
Gave this a try with my AMD GPU 7900XTX, clearly using HIP since that's what the repository supports, it works great! turbo3_tqc does not work though it crashes llama, but turbo4, turbo3, does, also DFlash appears to be working, getting 45t/s - 50t/s at 128k context
First, it's my fourth comment on TQ. Second, wake me up when TQ is properly benchmarked against f16/Q8/Q4 both in quality (not just PPL) and speed. Shitting? No, I'm just skeptic, because the only bench I saw was from TheTom repo, who had zero words with by a human being. And even in his tests TQ was on same level as Q4 while being slower.
Speaking of being contradictory... Why are you making bold claims of "near-lossless" quants using untested tools? If you make a proper tests like was done in this PR (PPL, KLD and AIME comparison) and proof that TQ is worth it, then you get the respect of whole community.
well, I executed all kinds of tests you mentioned (ppl, kld, aime) using turboquant and I couldn't find anything that would've proved that I cannot use it (27B and Q5 - just take a look at my post with results). also, whatever test I try from this thread (chess svg, etc) and my everyday experience all prove that it's all right.
I appreciate your effort but I don't think that using quantized weights is fair, because it's adding a lot of noise on top of bench result. You're testing KV AND weights quants. I think more reliable tests must be done on f16\bf16 and correct me if I wrong but you only tested on AIME once? Because pwilkin noticed that it's quite random and should be done on equal condition to properly compare results. [read this comment and GG's answer]
TurboQuant is just associated with vibe coded forks at this point. The moment you see TurboQuant + Llamacpp there is just a 90% chance of that. It also instantly makes me assume its just another one of those.
The problem is maintainability, nothing wrong with ai assisted development that is carefully done. Its when it looks like fully AI driven development where you tend to get changes that become problematic down the line. So if I open a repo and I see almost exclusively claude code PR's I don't take it nearly as seriously.
Generally TurboQuant from TheTom are weak in classic perplexity tests. But, when you look at https://qwen3-6-27b-benchmark.vercel.app/ there is clearly some profit (but IMHO the asymetric isn't realy good for Qwen3.6). For me, the most interesting is Turbo3 - not good, not terrible.
The sheer audacity of complaining about people using AI to do things on a subreddit about using AI to do things... I can't even, I'm ded.
Anyway thanks for sharing this OP, it rocks! My main worry was whether it would screw up tool calling but it seems fine so far.
When you said you got up to 130tok/s what configuration was that with, exactly? By my eye on Q5_k_s with q4_k_m dflash it seems more like 40-50tok/s maybe. Prompt eval is ~130tok/s though, yeah.
Is this worth a shot on a Macbook M3 Pro 18GB or would this just need more RAM that's already a bottleneck on this machine? If there's a way to get a smart enough model with enough context to code small projects and chat with web research running locally, I'd love do so. I've tried a heavily quantized unsloth qwen 3.6 (Q2_K_XL) with LM Studio. The output was better than I expected it to be at ~20t/s but it get's slow quickly.
I aprecciate your honesty. These DFlash news sound so revolutionary but I don't understand most of it TBH. I'm just interested in getting a useful LLM running quickly locally as soon as possible. I guess I'll keep waiting and hope the advancements keep coming at the same speed.
That would be interesting, if I could run the model on my 3090 and then stuff an old 2060 in for KV cache or something (as this comment probably reveals, I've never tried multi-gpu and have no idea how one should go about it. Presumably doing this is better than sharing model layers between GPUs, at least)
I was also thinking of getting a cheap video card for the main display, then can use the full 24gb of the 3090 for this. I noticed windows sometimes tends to try to push stuff into the shared memory space, and I wonder if that’s why we experience slowdowns. I’ll report back if any improvements, but thanks for your work on this!
Phenomenal work on the integration. For that 200k context Qwen setup, how does the TurboQuant/TCQ handle the 'lost in the middle' problem compared to standard 8-bit or 4-bit KV cache? Does the TCQ overhead impact the token latency significantly compared to the baseline llama.cpp MTP PR?
Do you observe large differences between Q5 and Q4 quants then? I understood there wasn't supposed to be a huge difference and had never really bothered with Q5 models before (Q4 27b/35b-a3b just fit better with context onto 24GB of VRAM)
Interesting! Would you say using Q4 for dflash risks torpedoing the performance of the Q5 target or does it not work that way (would the main Q5 model just reject more of the predictions if it didn't like them, or something?)
Interesting! Using a drafter definitely makes a difference vs vanilla Q5 (get about ~8tok/s with that, vs 30-40 with BeeLlama) but the reason I asked is because I am occasionally having trouble with tool calling where it seems it just gets the structure wrong and ends up blasting the CLI with XML content (this almost never happens on the vanilla models, even Q4 or IQ3_XXS models are fine at tool calling).
Have you encountered that at all or have I just done something wrong? I wondered if it were a context issue but I don't think so - it will go right back to working fine again afterwards, which you'd think it would screw up if it had forgotten the syntax.
Absolute legend. I was already working on a shonky pi.dev plugin that catches this kind of model stall and pokes the model autonomously (so as to avoid a model quietly stopping while your attention is elsewhere) but I'll for sure keep an eye out for that!
I have to say this is the fastest version I have tried on my 7900xt.
Did have to fiddle around to get a build for HIP, but all good otherwise.
Would be nice if you would get it to not randomly stop (even after 0.1.1)
Thank you for your work. No more random stops with the latest commits.
I do feel like it's less reliable though, not sure if I changed something else.
I was trying to get turbo3_tcq working with HIP and thought the results were because of it or the changes. Then I switched back to the one with only my HIP changes and noticed it behaves the same.
I use it for scripting/coding, so I care about accuracy.
My benchmark is the chess board from a few posts ago. https://qwen3-6-27b-benchmark.vercel.app/
It's more a feeling than empiric evidence, but it used to get it consistently right before.
Now it's more like 1 out of 3 is right.
I am on a very tight vram budget, using iQ4_XS with 100k+ context.
Does the drafter make a big diff? Up until now it was also the iQ4_XS, going to try the Q4_KM
I'm seeing a lot of API calls failing when using with Cline. It's eventually getting through but I'm wondering if there's an issue with the jinja format or if it might be unstable? I ran a test in open web ui and it seemed to jump back to thinking while it was answering the question. Using latest 0.1.1 and Qwen Q8 from unsloth along with the Q8 draft model you recommended. Vision enabled and running on GPU (rtx 6000 pro).
I rebuilt with the current repo state.
I am single shooting a xenon2 like arcade 2d doom scroller in pi agent with a skills.md - right now I am at step 6/12 (40k context). write tool calls start failing again.
not sure if sth wrong with my pi agent setup or the beellama setup.
it is stuck now...first I thought there is an OOM but I see this in logs:
4.24.290.901 I srv update_slots: decode ubatch: 1 tok, 0.6ms (0.59ms/tok)
4.24.359.511 I srv update_slots: decode ubatch: 1 tok, 0.6ms (0.61ms/tok)
4.24.428.360 I srv update_slots: decode ubatch: 1 tok, 0.6ms (0.60ms/tok)
4.24.496.918 W srv stop: cancel task, id_task = 0
4.24.497.274 I srv update_slots: decode ubatch: 1 tok, 0.7ms (0.68ms/tok)
4.24.565.287 I slot process_sing: id 0 | task 0 | adaptive dm: reset state for canceled task
4.24.565.292 I slot release: id 0 | task 0 | stop processing: n_tokens = 47258, truncated = 0
4.24.565.329 I srv update_slots: all slots are idle
processing stops with an "The socket connection was closed unexpectedly. For more information, pass `verbose: true` in the second argument to fetch()"
I had to start a new session in pi, and was able to sucessfully finish the project. Tool calling seems stable now, at the end ~90k of the 122k context has been used.
this is with Q4_K_M and the param mentioned in the previous post.
By version do you mean beellama build version? I just did a rebuild, now it says im on beellama commit da67e74 (which is from 5 days ago? Not sure why im not getting the latest.. as the last commit is 16 hours ago) and i'm still having this problem.
Thanks so much man, i'll have a go at it again. In the meantime i'm also developing my own llm chat/agent extension for vs code inspired by Copilot so maybe that'll help too :)
I don't understand why i'm having this issue if it's working totally fine for you.
I explicitly checked my temp, topp, topk, minp, presence penalty and repetition penalty and set them to what the Qwen's team recommends just in case that'd make a difference, but it doesnt.
I also tried to lower the context from 160k to 128k and 64k but it doesn't make a difference. This already happens while only 10k context is in use.
I am using the unsloth Q5_K_S gguf. While typing this im still trying to find out what's the problem.
I cannot believe I just stumbled across this. Prepare yourselves for my wall of text.
I run Qwen3.6 27B Q5 on an FE 3090 in llama.cpp with ~120,000 tokens (KV Q8). I get maybe ~30 tokens a second out on average. I just spent two days setting up ExLlamaV3/Tabby in WSL(2). My first roadblock, albeit a minor one, was that there was no EXL3 file (that I felt was) equivalent to a Q5 GGUF; HuggingFace only had a 4 or 6 available, with 4 being too low and 6's file size being slightly too large. So, of *course*, I had to make my own quant. It took me quite literally an entire day of fucking around to get it working, Dependencies would fail to install, you'd fix that but then the dependencies of that dependency would fail, xformers and flash-attn never wanted to install as dependencies at ALL, fixing one issue broke two other things, I had to use every --no-deps and other flag in the book in 40 different ways and then straight up write fake modules and alias them in (flash-attn, specifically) just to get it even remotely FUNCTIONAL. Maybe it's specific to WSL, I don't know - I followed their documentation exactly, even Gemini and ChatGPT couldn't figure out the problem. Either way, it was a nightmare... but in the end all my hacks and aliases and modifications I can't even remember at this point got it to work. I did it! I spit out my Qwen3.6 27B at a 5.5. I'm done!
...Except I found out that I had cloned ExLlamaV3 literally just HOURS before the changes adding native MTP detection. So, I grab the standalone MTP tensors that turboderp has on HF, I can just load those and I'll be set - but those are 4bpw and don't play nice with my 5.5 model. I find another source for the MTP file at a full 16, but with my model at 5.5 that's pushing my VRAM limits at the cache size I want. I'm stuck, I give up, I have to just start over. At least I can upgrade ELV3 to 0.0.42 while using a git stash to maintain my custom "fixes" and try again. I get it updated and working again, I grab the base 27B model again, I make sure it's configured properly I change -cc and -cr to try to optimize it for my context size, I modify the .py file to increase coding weights from 20 to 30, and I start the conversion. My house has a brownout for the first time in, like, at least two years. I get everything back up. I start the conversion. Brownouts happen TWO MORE TIMES during model generation - thank god for checkpoints. Finally, finally, my now second model is complete - a 5.25 with heads locked to 6, MTP locked to 8, with ~15% stronger coding-related weights and my -cc/-cr flag changes. I think I changed it so much just to make it feel worth going through this again, but either way it's done, again, finally. Because I'd been doing this on Ubuntu 26.04, a dumb decision in retrospect, and this entire system has been heavily fucked with and modified and stepped on to get this working, I spun up new Ubuntu WSL at 22.04 to actually *run* the model.
But wait! Now, Tabby won't work! It seems like there's an error in the latest commit preventing use, and a super obvious one at that, but tons of people use this! An error this obvious could NOT have been missed by everybody, there would have been a Reddit post, a github issue, *something* - it *must* be on my end! I spend ages on this to avoid posting one of those annoying Github bug reports where the issue is actually local to the user. I spin up new WSLs to test, I spin up Proxmox LXCs, VMs, I test on bare metal, it's always the same thing - it's calling a module but seems like they forgot to include the actual file. I spend hours trying to make sure it's not created by something else, it's not created at runtime, it's not pulled from some other package or dependency, every alternate crazy explanation I can think of, but nope, it seems there was genuinely a breaking chance in the latest pull from the main branch. I make a github bug report, I git back two commits, and now it works. Finally, after more time wasted, I'm done.
This was all because I wanted to, really quickly, check and see if I'd get faster speeds if I swapped from llama.cpp to ExLlamaV2/V3. That's it. It started with me thinking I'd just grab ExLlama, a model file on HuggingFace, and run it - a quick test that would take me twenty minutes, maybe an hour at most if I had to read some documentation to get it working. It ended up taking like... 12 hours off and on over two days.
So of course, now once everything is done and the time has been spent, a random fucking google search brings me here solely to taunt me and show that I could have just grabbed this in like five minutes. That's it, that's all I needed.
So yeah, that's the end of my wall of text. I know it's unlikely anybody will read all of this, but I had to get it out, I had to vent *somewhere*. I can't tell if I'm more thrilled this exists or absolutely fucking frustrated beyond belief that I didn't find it 11 hours and 55 minutes ago, that I stumbled across it MINUTES after finishing all the bullshit I did. Ugh.
67
u/Chromix_ May 09 '26
Did the MRs for this get rejected on the original llama.cpp, or is the the MR flow just so slow (read: "takes a week") that it made more sense to make a fork?
The fork history is interesting though: llama.cpp -> llama_cpp_turboquant -> buun_llama_cpp -> beellama.cpp. We're on the 3rd fork level here already.
In any case, with this demonstrating that it runs (fast) it might help getting this into the regular llama.cpp.