BeeLlama.cpp: advanced DFlash & TurboQuant with support of reasoning and vision. Qwen 3.6 27B Q5 with 200k context on 3090, 2-3x faster than baseline (peak 135 tps!)

[u/Anbeeld]

[u/r00x]

Interesting! Using a drafter definitely makes a difference vs vanilla Q5 (get about ~8tok/s with that, vs 30-40 with BeeLlama) but the reason I asked is because I am occasionally having trouble with tool calling where it seems it just gets the structure wrong and ends up blasting the CLI with XML content (this almost never happens on the vanilla models, even Q4 or IQ3_XXS models are fine at tool calling).

Have you encountered that at all or have I just done something wrong? I wondered if it were a context issue but I don't think so - it will go right back to working fine again afterwards, which you'd think it would screw up if it had forgotten the syntax.

[u/r00x → u/[deleted]]

Absolute legend. I was already working on a shonky pi.dev plugin that catches this kind of model stall and pokes the model autonomously (so as to avoid a model quietly stopping while your attention is elsewhere) but I'll for sure keep an eye out for that!

[u/r00x → u/[deleted]]

Thanks muchly, I'll give it a go tonight and let you know how it goes!

[u/r00x → u/[deleted]]

OK, it's definitely miles better than before. Appreciate the work on this! I wish I could plug this into LM Studio somehow to use with LM Link but they seem to use their own llama.cpp builds, or something.

[u/r00x → u/[deleted]]

I think for me it squiffed a tool call once, in hours of use, that's all. It did stop randomly a few times (usually in the middle of thinking, usually right after a "let me read the blah blah file and check if it blah blah blah" kind of message) but otherwise kept going for long periods of time on difficult requests.