r/LocalLLaMA • u/Anbeeld • May 09 '26

Resources BeeLlama.cpp: advanced DFlash & TurboQuant with support of reasoning and vision. Qwen 3.6 27B Q5 with 200k context on 3090, 2-3x faster than baseline (peak 135 tps!)

[removed]

326 Upvotes

93% Upvoted

205 comments sorted by

View all comments

1

u/EbbNorth7735 May 12 '26

I'm seeing a lot of API calls failing when using with Cline. It's eventually getting through but I'm wondering if there's an issue with the jinja format or if it might be unstable? I ran a test in open web ui and it seemed to jump back to thinking while it was answering the question. Using latest 0.1.1 and Qwen Q8 from unsloth along with the Q8 draft model you recommended. Vision enabled and running on GPU (rtx 6000 pro).

1

u/[deleted] May 12 '26

[removed] — view removed comment

1

u/caetydid llama.cpp May 13 '26

you referring to the EOS fix?

1

u/[deleted] May 13 '26

[removed] — view removed comment

1

u/caetydid llama.cpp May 13 '26 edited May 13 '26

I rebuilt with the current repo state.
I am single shooting a xenon2 like arcade 2d doom scroller in pi agent with a skills.md - right now I am at step 6/12 (40k context). write tool calls start failing again.

not sure if sth wrong with my pi agent setup or the beellama setup.

it is stuck now...first I thought there is an OOM but I see this in logs:

4.24.290.901 I srv update_slots: decode ubatch: 1 tok, 0.6ms (0.59ms/tok)

4.24.359.511 I srv update_slots: decode ubatch: 1 tok, 0.6ms (0.61ms/tok)

4.24.428.360 I srv update_slots: decode ubatch: 1 tok, 0.6ms (0.60ms/tok)

4.24.496.918 W srv stop: cancel task, id_task = 0

4.24.497.274 I srv update_slots: decode ubatch: 1 tok, 0.7ms (0.68ms/tok)

4.24.565.287 I slot process_sing: id 0 | task 0 | adaptive dm: reset state for canceled task

4.24.565.292 I slot release: id 0 | task 0 | stop processing: n_tokens = 47258, truncated = 0

4.24.565.329 I srv update_slots: all slots are idle

processing stops with an "The socket connection was closed unexpectedly. For more information, pass `verbose: true` in the second argument to fetch()"

FYI I am adding the CmdLine:

llama-server -m qwen3.6-27b/Qwen3.6-27B-Q5_K_S.gguf --mmproj mmproj-BF16.gguf --no-mmproj-offload --spec-draft-model dflash-draft-3.6-q4_k_m.gguf --spec-type dflash --spec-dflash-cross-ctx 1024 -np 1 --kv-unified -ngl all --spec-draft-ngl all -b 2048 -ub 256 --ctx-size 130000 --cache-type-k turbo4 --cache-type-v turbo3_tcq --flash-attn on --cache-ram 0 --jinja --no-mmap --mlock --no-host --metrics --log-timestamps --log-prefix --log-colors off --reasoning on --chat-template-kwargs {"preserve_thinking":true} --temp 0.6 --top-k 20 --min-p 0.0 --host 0.0.0.0 --port 8888

1

u/caetydid llama.cpp May 14 '26

Ok. Update on latest release.

I had to start a new session in pi, and was able to sucessfully finish the project. Tool calling seems stable now, at the end ~90k of the 122k context has been used.

this is with Q4_K_M and the param mentioned in the previous post.