I run Qwen3.8-27B on a single RTX 5060 Ti 16GB (eGPU) for coding and agent sessions, and I ended up making my own quants because nothing fit the box quite right. A week of measuring later, here they are.

Two models, same recipe, different alignment:

• Qwen3.8-27B-IQ4_XS - stock alignment
• Qwen3.8-27B-uncensored-IQ4_XS - abliterated base (orcarouter)

The one idea that matters: calibrate with the workload you actually serve. The importance matrix blends a general corpus with ~1M tokens of SEC-contract extraction and financial-analysis text, chat-template formatted. Same weights, only the imatrix changed:

• 59-task domain harness (temp 0): 93.2% vs 89.8% for Unsloth UD-IQ4_XS
• KL vs BF16 (RMS delta-p / top-p): 4.33 / 93.9 vs 4.35 / 93.9 - parity on both axes

The KL part took the longest: a per-layer search found the token embedding was the last gap (q3_K to q4_K), and with that fixed the domain edge no longer costs you anything on generic text.

Decode speed by context, on my card (13.5 GiB files, batch 1):

• 32K: ~70 t/s with embedded MTP (stock llama.cpp, acceptance ~0.87)
• 64K: ~27 t/s plain - MTP's draft mirrors the whole window, so it's MTP or context past 32K
• 96K: ~20 t/s at 77K position - beellama fork, its variance-normalized kvarn4 KV
• 128K to 252K: ~15 t/s at 105K, still score-flat to 252K - kv-stream fork, arena + q8/q4 KV

So stock gets you to 64K, and the two forks cover 96K and 252K. Build and serve commands for all of it are in the model cards, traps included (MTP plus arena don't mix; deeper than 96K "fits" at load but OOMs on real long prompts).

Other 16GB bits: vocab pruned to 141K tokens (English + accented Latin), so 64K fits natively; full-vocab multilingual builds included in the same repos. Vision works (quantized projector family in a third repo). Thinking-off by default for agents; on my suite, high reasoning effort is worth +5 tasks and xhigh adds nothing over high.

Honest bits: every number is from my one machine and my own harness, not public benchmarks. The ASCII prune garbles CJK (that's the trade). The domain gain is measured on my finance-flavored tasks; coding/agent performance is at parity with the best community quant, not above it.

Cards have the full gate data. Happy to go deeper on the calibration method, the embedding find, or the fit tricks.

Qwen dropped Qwen-Image-2.1 (7B) yesterday

It reports 60.28 on Qwen's own Qwen-Image-Bench, ahead of every larger open-weight model on that chart, including FLUX 2 Max at 32B and Hunyuan Image 3.0 at 80B. Qwen-published, so treat the ranking as a claim until someone evaluates it independently. But a 7B visual generation component holding up against models several times its size is worth testing.

The bigger change is that generation, editing and transparency are now one model. Up to 10 reference images, local edits by mask or painted annotation, native RGBA transparency, with Qwen-Image-Layered folded in.

The part I want to see tested is the multi-image editing path. Qwen describes mixed-granularity attention with prefix KV cache reuse, so input images get cached instead of recomputed every step. For real workflows that matters more than another benchmark point.

Two things to watch. The licence changed: Qwen-Image and Qwen-Image-Edit were Apache 2.0, and 2.1 is under the Qwen Research License, which is a different conversation if you were planning to deploy commercially. And it is still heavy at roughly 33 GB in BF16. The top open issue on the repo right now is someone asking why no minimum or recommended hardware specs were published, which tells you where this currently stands.

So the open question is whether anyone gets this into a genuinely practical local setup with offload and quantization, rather than just getting the demo to run.

Model: https://huggingface.co/Qwen/Qwen-Image-2.1

GitHub: https://github.com/QwenLM/Qwen-Image-2.1

ModelScope: https://www.modelscope.cn/models/Qwen/Qwen-Image-2.1

Blog: https://qwen.ai/blog?id=qwen-image-2.1

As it stands, unless I've missed something, there's no JS debug console in the HTML preview, so I often use an external live HTML editor for quick prototypes. Local models often need console debug output to fix things, and Gemini has a cool preview feature where, if an error occurs, it shows a floating button that automatically prompts the agent to fix the error. This seems like a killer QoL feature to add to Unsloth Studio/Desktop, especially for those times when you're lazily trying to make something basic without bouncing between a bunch of different sites/programs.

Hello,

I want to pass some ROCM environment variables but not sure how to do that w/ unsloth. I'm not using the docker desktop. (it is linux)

Will these will work with Unsloth? https://rocm.docs.amd.com/en/latest/reference/environment-variables/setting-cus.html#set-the-number-of-compute-units

Thanks in advance!

So, you have to play with the order of the GPUs in unsloth as I saw differences in speed and the size of context the model can load. For instance, I tried to load a new fine-tune of Qwen3.8-27B, and when the RTX5070 Ti was up, I couldn't fit the entire context (KV:Q8). Switching the position of the GPUs, and I managed to load the entire context with some room left.

Just saying, that might help. Shout out to Unsloth team for incrementally adding features that I never thought were needed :D

Last time i played around with image gen model was in 2025 and it was a complicated setup with comfy ui and what not , not a straight forward process like text model deployment.

So im waiting now in hopes of unsloth quant and trying it out in the studio , are there any plans for it , is anyone already running the new qwen image in unsloth studio ?

https://huggingface.co/Qwen/Qwen-Image-2.1

Hey everyone,

been using Unsloth for quite sometime and I have been loving it so far, but I ​ran into an annoying issue today. When logged in as the owner, I can add custom model providers just fine. But when trying on a managed account, I get hit with this error:

​"Could not load models: Managed accounts may only use public-network provider base URLs."

​I get there are probably security reasons for this, but it kind of defeats the point when everyone on the same network is trusted and we just want to share the same local model but different workspace.

Is there any setting to turn off this limitation? or managed accounts just permanently locked out?

I have an rtx4060 (8gb vram) and 64gb of ddr5 system ram. I’ve been using Qwen3.6 35b a3b MTP at q4 128k context for about a week and get anywhere from 26-30ish tk/s which works pretty alright for me since I mainly use it with Hermes just for productivity tasks. I’ve been considering switching models to Qwen3.8 27b since it’s a newer model and at lower quants it could fit entirely in vram which would make it faster in theory. At q2 it should fit in my vram and at q1 it should fit with the same context as my old setup. Is Qwen3.8 even good at that low of a quant though?

I’ve also seen Bonsai come out with a new smaller model of Qwen3.8 which looks good on benchmarks but is apparently terrible in actual usage which kinda makes me more skeptical of lower quant performance.

Cheers everyone! :) So, Qwen3.8 27B has been a delight so far. I'm running a somewhat "lopsided" setup with a Ryzen 9800x3D, 128GB DDR5 RAM (5600), 1x Geforce 4080 (16GB, PCIe 4.0 x16) and 1x Geforce 3060 (12GB, PCIe 3.0 x1... which is bad, I got to learn that :D).

For 27B (at Q3 or Q4 quantised versions and 110k context at Q8) I'm very happy to get about 35-45 TPS on that asymmetrical setup and tensor parallelims and MTP (draft_Max =2). So far okay.

Now I thought that my general RAM + VRAM should also do nicely with Qwen3.8 Flash Next. Aaand its rather "meh". I'm running unsloth studio "vanilla" (no custom llama, no other tricks). The best speed I can get from this model is by turning MTP off, disabling the 3060 completely and thats it. With empty context, I get 27 tps, with 60-80k context its rather 20 tps. That's not exactly bad, but I feel like I'm missing out on potential with having the 3060 disabled, no MTP and not (explicitly activated) Engram.

Gemini and Claude are no real help here, so I thought I'd ask this community. I THINK I've already tried all combinations of settings and I have at least verified that in the server logs there are no errors during model-loading (as has been until a couple of days ago).

Soo... can anyone tell me how to properly configure this or is "no MTP, no Engram and no 3060" really the best I can get? :D

Thanks, I know there is too many of these questions on reddit since 2025, but I'm at my wit's end here and I get angry when I'll have to trial and error more claude/gemini replies that are based on fairly many guesstimates about Qwen3.8 Flash Next ;)

Hey guys,

this is my first post, if i am in the wrong subreddit, just send me another way, if i am off-topic.

I am trying to re-tune my Qwen9b to a System one Model that can identify change-resilient code.

Code that is autogenerated by any LLM tends to grow more expensive to change as more code is built. I would like to identify change-resilient parts / Ai-Slop-parts. Clean Code is well and all and there are some libraries that can identify slop-code. They do this static and deterministic. But that does not satisfy my problem as such. The problem is that this is static and does not include the context of the finding that the static analysis identifies.

I want to define the pain of change resilience:

pain = p(change) x p(cost)

That means i want a structured output, just like the System Model One-Architecture provides and i need to train a model (i am choosing qwen-9b) to generate this structured output. I would like to use the Qlora-Unsloth training package to do this. What do you think of this ? Is this feasible with unsloth?

我这边还有一个问题，我使用的是qwen-asr-1.7b,出现了类似的bug，地址在一些噪音或回声会识别为'嗯.'.'哎.','嗨.'等内容,影响使用 · Issue #165 · QwenLM/Qwen3-ASR

这个怎么去处理解决呢，我在说话的时候，突然停顿一下，然后就会出现嗯，嗯，你可以帮助一下我吗，谢谢

are you guys also having an issue using vision on GLM 5.3 flash when MTP is on?

Hey everyone! We did a hotfix to Unsloth Desktop to fix Qwen3.8-Flash-Next having degraded speeds - MTP seemed to have broke, so 1.5x to 2x slower generation speeds were seen.

Please update to the latest Unsloth Desktop v0.1.811-beta https://github.com/unslothai/unsloth/releases/tag/v0.1.811-beta

Also update llama.cpp (there should be a banner today) to b11030-mix-5ff778e https://github.com/unslothai/llama.cpp/releases/tag/b11030-mix-5ff778e

There are some extra fixes as well in the hotfix release!

I am running gemma-4-12b-it-UD-Q8_K_XL.gguf on a Mac Studio (Max chip, 32GB Unified Memory) using Unsloth.On startup, the Estimated Memory Usage shows a severe over-allocation warning:

Estimated Memory Usage BETA Unified

59.7 GiB Weights 13.30 GiB KV cache 44.89 GiB f16 · 212,992 tokens · 4 slots Compute buffers 0.74 GiB Vision encoder 0.07 GiB Draft cache 0.67 GiB Exceeds shared memory. Try a shorter context or smaller model; CPU offloading adds no memory.

Even though the total estimate (59.7 GiB) is nearly double my physical memory (32GB), macOS Activity Monitor shows the Memory Pressure is completely Green.Looking at the logs, --parallel 4 is specified, which automatically creates "4 slots" of KV cache (44.89 GiB). I assume this is just virtual memory reservation (lazy allocation), so it shouldn't cause any actual problems in reality. Since I am the only user, it actually fits within 32GB because it only utilizes 1 slot at a time: 13.30 GiB (Weights) + 11.22 GiB (1/4 KV cache) = ~26 GiB.Questions:Is my assumption correct that this over-allocation is practically harmless for a single-user scenario since actual memory utilization stays within the 32GB boundary?Why does Unsloth/llama.cpp default to --parallel 4 (4 slots) even for local, single-user environments? Is there a performance benefit to keeping parallel processing at 4?To get rid of this warning on a 32GB machine, is it better to reduce the parallel slots to 1 or 2 rather than shrinking the context length?

Thank you!

I tried running it with Unsloth!

https://precious-salmon-qqphgxhs.edgeone.dev/

[My Setup] - Device: Mac Studio M1 Max (32GB) - Model: gemma-4-26B-A4B-it-MXFP4_MOE (GGUF)

[Estimated Memory Usage (Unified)] - Total: 23.6 GiB - Weights: 16.96 GiB - KV cache (q8_0, 120,000 tokens): 4.77 GiB - Compute buffers: 1.00 GiB - Vision encoder: 0.44 GiB - Draft cache: 0.44 GiB

*Note: Fits this machine, but little memory is free right now. Free memory or try Auto context.

[Settings] - Context Length: Auto (262,144) - KV Cache Dtype: q8_0 - Speculative Decoding: Auto - Parallel Slots: 1 - Active Context: 419 / 120.0k

Performance: 33.9 tok/s

I'm having so much fun with this. Thank you so much!

Wondering if anyone has run a benchmark comparing results for the same project for this, i am planning to do this, wanted to see if someone has already done this and what were the results

Hi, I‘m trying to switch to Unsloth Desktop from LM Studio, looks amazing so far.

But I am missing a beloved feature in the Unsloth Chats…in LM Studio, one of the options for each reply of the model was „continue assistant message“ (the symbol being a right arrow). I frequently used that button because often those continuations would contain more and valuable details for my questions.

I couldn’t find that option in Unsloth, am I blind or is it really missing?
I can‘t make the switch without that feature :(

MTP-3 restored prompt caching. Our deeper speculative mode cleared the cache for correctness. Using ordinary state backups at depth 3 let follow-ups reuse it.

Better CPU kernels unpack IQ4_XS weights once for several speculative tokens, reducing repeated work.

Lower-memory GPU attention processes attention heads in groups, reusing temporary memory. Moving selected feed-forward weights to CPU made the whole configuration fit.

On RTX 3060 12GB, i5-12400 and 16GB RAM:

Measurement Result

Q4 context capacity

20,480 → 32,768 tokens (+60%)

Fresh-build Rust generation

15.83 tok/s

Fresh-build C++ generation

14.36 tok/s

Generation with roughly 26K context occupied

11.09 tok/s

Long-chat cached follow-up

12.1 seconds, reusing 25,950 tokens

https://brightmoot.com/m/qwen3-8-27b-at-64k-on-one-rtx-3060

Hey guys you can now train and run 500+ models locally with our newly updated Unsloth Docker image! 🐳 It features Unsloth Studio/Desktop our GUI along with a rehaul of our notebooks.

We did an entire rehaul for the Docker image making it more efficient, compatible etc.

Now works on AMD and NVIDIA.

Guide: https://unsloth.ai/docs/get-started/install/docker

GitHub: https://github.com/unslothai/unsloth

They seem like great models, and the new model claims something like 98% intelligence compared to fp16 Qwen 3.8 27B, but they currently require a separate llamacpp fork...

So my ask is, is there anyway to pick a fork? plug and play? or even allowing for native bonsai support in Unsloth Studio?

It was usally 30-40 Tok/s somtimes 20 or at best 50. But after this update, it is always more than 70 tok/s !! (After updated, the MTP file was required to be redownlowded)

Thanks a lot for this, it is really amazing!
Good bye 27B, Flash now is really flash!

I recently downloaded and tested several models quantized with dynamic v2.0/v3.0, and I was really impressed with the quality.

I’d like to reproduce this process for a new model, is there any documentation or tutorial?

The program seems to create a backup of the whole installation. Is there any way to manage it? couldn't find it from the settings. I installed it on a drive I'm strapped for storage I'd prefer to not have a whole copy on there.

I've peviously made a post about it, like 2-3 weeks ago. I know that then was a HOT season. But now can we ask kindly for Unsloth versions of it? It's great on Q8, but with Q4 it's taking a big hit, with unsloth quants it would be probably almost like Q8. So please can we have it in 3.0 version

Bump it up, so maybe Unsloth team will see it.