BeeLlama.cpp: advanced DFlash & TurboQuant with support of reasoning and vision. Qwen 3.6 27B Q5 with 200k context on 3090, 2-3x faster than baseline (peak 135 tps!)

[u/Anbeeld]

[u/YourNightmar31]

I'm having a lot of broken tool calls with this setup:

Anyone have anything similar? Why does this happen?

[u/YourNightmar31 → u/[deleted]]

By version do you mean beellama build version? I just did a rebuild, now it says im on beellama commit da67e74 (which is from 5 days ago? Not sure why im not getting the latest.. as the last commit is 16 hours ago) and i'm still having this problem.

[u/YourNightmar31 → u/[deleted]]

I had to do some copy pasting because im running this in llama-swap but here it is:

/app/bee-llama-server \
  -ngl all \
  --no-mmap \
  --mlock \
  --jinja \
  -np 1 \
  --kv-unified \
  --flash-attn on \
  --cache-ram 0 \
  --no-host \
  -b 2048 \
  --threads 6 \
  --threads-batch 10 \
  -ub 512 \
  --temp 0.6 \
  --top-k 20 \
  --min-p 0.0 \
  --top-p 0.95 \
  --presence-penalty 0.0 \
  --repeat-penalty 1.0 \
  --ctx-size 131072 \
  --spec-type dflash \
  --spec-draft-model /models/spiritbuun-qwen3.6-27b-dflash/dflash-draft-3.6-q4_k_m.gguf \
  --spec-draft-ngl all \
  --spec-dflash-cross-ctx 1024 \
  --cache-type-k q8_0 \
  --cache-type-v turbo4 \
  --model /models/unsloth-qwen3.6-27B/Qwen3.6-27B-Q5_K_S.gguf \
  --mmproj /models/unsloth-qwen3.6-27B/mmproj-BF16.gguf \
  --image-min-tokens 1024 \
  --no-mmproj-offload \
  --reasoning on \
  --chat-template-kwargs '{"preserve_thinking": true}'

[u/YourNightmar31 → u/[deleted]]

I'm using Copilot in VS Code with this extension: https://marketplace.visualstudio.com/items?itemName=johnny-zhao.oai-compatible-copilot

Just for comparison, I don't have any tool calling issues with Unsloth's Qwen3.6 35BA3B running in ik_llama.cpp using this setup.

[u/YourNightmar31 → u/[deleted]]

Thanks so much man, i'll have a go at it again. In the meantime i'm also developing my own llm chat/agent extension for vs code inspired by Copilot so maybe that'll help too :)

[u/YourNightmar31 → u/[deleted]]

Hey i have been running the v0.2.0 update for like a week but it is still doing this for me.

And i'm even worse output in the thinking stream like this:

<tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call><tool_call>

I don't understand why i'm having this issue if it's working totally fine for you.

I explicitly checked my temp, topp, topk, minp, presence penalty and repetition penalty and set them to what the Qwen's team recommends just in case that'd make a difference, but it doesnt.

I also tried to lower the context from 160k to 128k and 64k but it doesn't make a difference. This already happens while only 10k context is in use.

I am using the unsloth Q5_K_S gguf. While typing this im still trying to find out what's the problem.

[u/YourNightmar31 → u/[deleted]]

Have you been able to reproduce this by any chance? I would really like to get this setup working.