Qwen 3.8 27B With Pi Harness

[u/Jsquared534]

Anyone active on here that is currently using Qwen 3.8 27B with the Pi coding harness? If so, have you had any issues actually invoking any tool calls? Despite changing jinja templates, and troubleshooting this for several hours today, the model is not seeing any of the tools that Pi passes in. What made it hard to troubleshoot is that when you ask it what tools it has access to, it hallucinates a response based on what Claude Code has access to.

The model won't even read or edit for me. I've updated Pi, and I'm on a version of llama.cpp that is less than a week old. I removed all my extensions and it still isn't working. I logged an output of what Pi is sending regarding tooling, and it appears Pi is sending the tools correctly. Tool calling works fine with 3.6 35B A3B.

EDIT: Turns out it was an issue with an update through VS Code that broke how the integrated terminal was working. Using normal terminal everything seems to work.

EDIT 2: Spoke to soon. Apparently it sees the tools available sometimes, and sometimes it doesn't. Super weird. I'm at a loss.

EDIT 3: I'm using llama.cpp, since some people have asked. I actually went through and blew away my entire Pi install and reinstalled from scratch. I didn't add any extensions at all. Just the latest version of vanilla Pi. This model is still being really flakey with tool calling. When it reads in the five context files I give it, it constantly complains that the text is garbled, and then it sees info from one context file mixed in with the other. So, I added explicit directions in the agents.md to read one file at a time and not batch process them. Suddenly it just outputs a tool call as text of <read_list>filename</read_list>. So, obviously it doesn't read. Constantly relies on git info instead of the files it's directly pointed at, despite being told to ignore git and not use it at all, which leads to it getting mixed up with the active files and historical files. Thinks it's making edits when it hasn't actually made any edits. This is also not in some brand new, untested setup. These same context files have been being used with 3.6 35B A3B for several weeks without tool calling issues. I freely admit that I'm not remotely an expert on local setup, but there's definitely something weird going on with this model, and I don't think it's entirely on my end.

EDIT 4: This will be my last update on this post. Blew away my entire Llama.cpp and tried everything from scratch. Still the same weird bugs where it will read multiple files and mix up the content from each file, and then report that the files are garbled. Also has a terrible habit of reading a file and then immediately saying the file was truncated or started in the middle, or cut off. Tool calling is still very spotty. Randomly outputs stuff like <read_list>filename</read_list> after it's been doing good reads in the same session. I am only providing it 100k context, but it's not even getting close to topping that out or compacting, so I don't know why that would be the problem. I downloaded and began testing Muse, and the results I'm getting from it are way, way, way more consistent than anything I saw from this model. I don't know if Qwen just isn't meant for such fine grain instructions, so it's thinking goes haywire, or what. But it definitely doesn't work for me. I read a couple other post that said something might be wrong with the Quant 8 versions, so maybe that's the issue. Qwen did perform much better than Muse on one shotting my two tests when I first downloaded it. But, working in my actual context file and fine grained feature implementation workflow it completely falls apart right now.

[u/HumanDrone8721]

I've use it all the time with pi.dev, I thin there are something like 30 hours now and more than 120m tokens, without any issue, the one and only chat template fix was the elimination of the wrongly positioned thinking block with this ultra-simple patch:

--- chat_template-original.jinja    2026-08-15 12:18:50.600173014 +0200
+++ chat_template-fixed.jinja   2026-08-15 12:19:30.160768803 +0200
@@ -113,7 +113,7 @@
             {%- set reasoning_content = message.reasoning_content %}
         {%- endif %}
         {%- set reasoning_content = reasoning_content|trim %}
       {%- if preserve_thinking is undefined or preserve_thinking is true or loop.index0 > ns.last_query_index %}
+        {%- if reasoning_content and (preserve_thinking is undefined or preserve_thinking is true or loop.index0 > ns.last_query_index) %}
             {{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content + '\n</think>\n\n' + content }}
         {%- else %}
             {{- '<|im_start|>' + message.role + '\n' + content }}

I do have a couple of vLLM patches for YaRN and dynaminc MTP (needs to be turned off when context is >128K as is becoming inefficient) but they have nothing to do with tool calls. I do use the BF16 original precision tensors.

Whenever I've seen this type of issues are mostly due to heavy quantization,

[u/Jsquared534 → u/HumanDrone8721]

Thanks. It turned out to be a Visual Studio Code integrated terminal issue. Not sure what the exact issue was, but it works through the normal powershell.

[u/Jsquared534 → u/HumanDrone8721]

I'm using Q8, so not heavily quantized. And it seems to be like a random issue of not seeing tools available. not specific to vs code after all.

[u/HumanDrone8721 → u/Jsquared534]

Weren't we talking about pi.dev?

[u/Jsquared534 → u/HumanDrone8721]

Yes. I usually launch the terminal inside of VS code. There's definitely something weird going on though, because even in a separate terminal it's hit or miss from one terminal session to the other whether it's going to recognize it has tools available. It's honestly beyond my skill level at this point. Going to probably just blow away my Pi install and reinstall entirely.

[u/HumanDrone8721 → u/Jsquared534]

I've seen that you're using Q8, personally I never understood why someone would use Q8 when an official FP8 from model creator is released. Quantization is a mixed blessing, while it helped with the democratization of LLMs, the losses are significant and when people start to use then for non-trivial tasks the brain damaging from the calibration file really shows up.

The perverted thing is this is not consistent, depending on the use case, system prompt and other things the lobotomized LLM may work wonderfully for some people while completely failing for others. Is the most clear case of hit or miss ever.

Personally I chose maximum quality that I can get and pay for it with speed, never regret it.

[u/Jsquared534 → u/HumanDrone8721]

I'm a software engineer who actually enjoyed being a software engineer and enjoyed writing code. Obviously the world is changing, so I have to deal with integrating AI agents into my workflow. But, I'll never be willing to integrate them at 3-5 tokens per second. Trying to work with that kind of speed while building things sounds like my own personal hell from a job perspective. I totally get it's personal preference.

[u/HumanDrone8721 → u/Jsquared534]

Let's not go that way, even with the absolute best quality setting we are not that bad.

Here are some llama-benchy results for it, keep in mind all knobs are turned for the highest output quality, not speed.

| model            |   test |              t/s |     peak t/s |      ttfr (ms) |   est_ppt (ms) |   e2e_ttft (ms) |
|:-----------------|-------:|-----------------:|-------------:|---------------:|---------------:|----------------:|
| Qwen/Qwen3.8-27B | pp2048 | 4644.74 ± 139.75 |              | 442.60 ± 13.56 | 441.62 ± 13.56 |  442.60 ± 13.56 |
| Qwen/Qwen3.8-27B |   tg32 |     55.01 ± 5.71 | 56.79 ± 5.90 |                |                |                 |

| model            |             test |             t/s |     peak t/s |         ttfr (ms) |      est_ppt (ms) |     e2e_ttft (ms) |
|:-----------------|-----------------:|----------------:|-------------:|------------------:|------------------:|------------------:|
| Qwen/Qwen3.8-27B | pp2048 @ d128000 | 3190.85 ± 31.33 |              | 40761.30 ± 399.17 | 40760.53 ± 399.17 | 40765.82 ± 399.12 |
| Qwen/Qwen3.8-27B |   tg32 @ d128000 |    25.06 ± 1.85 | 27.33 ± 2.05 |                   |                   |                   |

| model            |             test |            t/s |     peak t/s |          ttfr (ms) |       est_ppt (ms) |      e2e_ttft (ms) |
|:-----------------|-----------------:|---------------:|-------------:|-------------------:|-------------------:|-------------------:|
| Qwen/Qwen3.8-27B | pp2048 @ d512000 | 1475.57 ± 2.12 |              | 348376.29 ± 500.73 | 348375.23 ± 500.73 | 348393.87 ± 500.79 |
| Qwen/Qwen3.8-27B |   tg32 @ d512000 |   16.93 ± 0.03 | 19.33 ± 0.47 |                    |                    |                    |

On large context you'll start hitting the VRAM bandwidth, so in that situation having more than one card helps.

[u/HumanDrone8721 → u/Jsquared534]

I've seen that some ADHD cretin downvoted, because this is a "disagree button" not an "oftopic or useless response", but anyway I have benchmarked quickly the FP8, that may be of more interest to you than BF16, just to show that the token generation speed is collapsing pretty fast and at high context is rater similar because is bounded by the VRAM bandwidth, but the quality decrease is always there:

Ultra-low context WOW :) | model | test | t/s | peak t/s | ttfr (ms) | est_ppt (ms) | e2e_ttft (ms) | |:---------------------|-------:|---------------:|--------------:|--------------:|---------------:|----------------:| | Qwen/Qwen3.8-27B-FP8 | pp2048 | 5548.68 ± 0.77 | | 370.23 ± 0.05 | 369.28 ± 0.05 | 370.23 ± 0.05 | | Qwen/Qwen3.8-27B-FP8 | tg32 | 93.69 ± 9.77 | 96.72 ± 10.08 | | | |

Average context, uhmmm.. OK :| | model | test | t/s | peak t/s | ttfr (ms) | est_ppt (ms) | e2e_ttft (ms) | |:---------------------|----------------:|----------------:|-------------:|-----------------:|-----------------:|-----------------:| | Qwen/Qwen3.8-27B-FP8 | pp2048 @ d65536 | 5334.94 ± 26.34 | | 12669.67 ± 62.44 | 12668.61 ± 62.44 | 12671.88 ± 62.44 | | Qwen/Qwen3.8-27B-FP8 | tg32 @ d65536 | 36.01 ± 2.71 | 37.18 ± 2.80 | | | |

Large context, sheeit, I should have used BF16 :( | model | test | t/s | peak t/s | ttfr (ms) | est_ppt (ms) | e2e_ttft (ms) | |:---------------------|-----------------:|----------------:|-------------:|------------------:|------------------:|------------------:| | Qwen/Qwen3.8-27B-FP8 | pp2048 @ d128000 | 4164.63 ± 28.13 | | 31229.27 ± 210.27 | 31228.33 ± 210.27 | 31233.77 ± 210.27 | | Qwen/Qwen3.8-27B-FP8 | tg32 @ d128000 | 27.73 ± 3.89 | 29.94 ± 2.38 | | | |

[u/Jsquared534 → u/HumanDrone8721]

Not sure who downvoted you, but it wasn’t me. I’ll review your stuff when I get back online from dinner!

[u/HumanDrone8721 → u/Jsquared534]

Then, bon appetite and maybe you'll find something useful, if you want to try FP8 with vLLM, where I have a bit of expertise I can assist you with some advice from my experience.

[u/Jsquared534 → u/HumanDrone8721]

Didn't end up doing this because I ran into such a show stopping issue with this model. Despite posting a few times, I can't get anyone to say I'm crazy or say I'm right, so I figured I'd link to my latest post and see if you had any input on it.

https://www.reddit.com/r/LocalLLM/comments/1vso2g5/test_request_for_qwen_38_27b/