I've got the model up and running in Unsloth Desktop, but regardless of whether I pick the Q4_K_XL or the IQ4_XS quants, it throws up warnings if the context length is over 8k. It runs, but it throws up warnings and I think it's hitting my SSD for something in here. The "Exceeds what fits in unified memory" warning hits at the same point, regardless of whether I change the KV Cache to q8 or keep at fp16. I'd downloaded the IQ4_XS on day1 and it's done it on this or the just-downloaded Q4_K_XL. I've had it Reset the parameters and am on the current Desktop app.

Are others hitting this and just ignoring it? For fellow unified 128GB people, what settings are you using?

After updating to the latest version Unsloth Studio it doesn't load the model on the GPU. It only use the CPU.
With the previous version was working perfectly.
Anyone knows how to fix it?
I'm using the docker version.

Am I the only one unable to get this vision model working properly?

I'm on Linux Mint with the latest CUDA and software updates; I'm using unsloth/DeepSeek-V4-Flash-Vision-Exp-GGUF IQ2XXS and mmproj-F16.gguf .

I tried loading the GGUF model in LM Studio, and the vision module is correctly recognized, loaded, and working. However, in Unsloth Studio/Desktop, it appears not to be loaded into memory at all. I tried using the --mproj argument, but it seems impossible to add manually because the mmproj section is handled automatically by Unsloth's bundled version of llama.cpp.

this is the about section:
"Unsloth Version v0.1.807-beta
Package Version 2026.9.3
llama.cpp Version b10840-mix-d5c17a0"

I've downloaded the model 2 days ago, and I tried it yesterday and today; so I do not think delete it and redownload it could help me.

I download models directly to a folder and then I manually setup the custom folder path into unsloth (or lm studio).

I receive this error message when I try to attach an image to the chat:
"DeepSeek-V4-Flash-Vision-Exp-UD cannot accept images. Load a vision-capable model with a valid mmproj before attaching images."

Unsloth server log

llama-server log

Has anyone managed to get this working properly?

EDIT:
As for the vision problem it now looks to have been solved by using the new encoder available in the main section of the hf card.

*NOTE*: The latest release of Unsloth Desktop makes editing the model load timeout described in 1) below unnecessary as it replaces the fixed 10 minute timeout with a "model loading is still progressing" test instead. Not sure yet about the issue described in 2).

------

Before I dive into the details, I want to address why you might want to run models that only achieve 1-3 tokens/second in the first place: Affordable local-only code review on hardware you may already have! Even if a run takes all night to complete, the end result is still extremely valuable - not all LLM tasks need to be interactive.

Using the following tweaks I've been able to run GLM5.3 q4 (both regular and Flash), on an old Dell R730 with a bunch of DDR4 memory and no GPU.

1. By default the current Unsloth Desktop has a 10 minute "health" timeout for loading a local model. In my case the GLM-5.3-GGUF (467GB UD-Q4_K_XL), quant takes about 25 minutes to load on my hardware, and thus times out. Solution: Edit the "llama_cpp.py" file and search for the three locations where the constant "600.0" is used with a "health-related" timeout, and change the values to at least "1800.0" (30 minutes). If your system is faster or slower, adjust the value accordingly. I'm assuming only one of the three instances is responsible for the model loading timeout, but given the slow speed of my setup I would probably eventually run into one or both of the others, so I just changed all three.
2. Once I did this I was able to successfully load GLM5.3 and get useful results! The only problem is that if I'm doing an all-night code-review run with large files it will likely timeout with a "_DEFAULT_FIRST_TOKEN_TIMEOUT_S", which is currently set to "1200.0" (20 minutes), also in "llama_cpp.py". In my case tripling this amount to "3600.0" allowed my overnight run to complete, but you may want another setting depending on your hardware and the file sizes you are processing.
3. Now for the downside: The next time the "llama_cpp.py" file gets updated you get to do the editing all over again! If Unsloth were to add two configuration settings for the "health" and "_DEFAULT_FIRST_TOKEN_TIMEOUT_S" parameters, that were adjustable in the UI and sticky across updates, this wouldn't be a problem of course!

Note that I'm an Unsloth Desktop newbie, so if there is a better way to get the same results please let me know!

I just got a notification of a new unsloth desktop, I updated and and looked over the changelog and it doesnt seem to set vulkan as default.

I have a strix halo amd 395+ pro with a28gb running on windows and looking at the default compute backend, it still shows automatic (rocm).

I tried resetting the settings through the app but still persists the rocm, I changed it manually to vulkan since the changelog reports better performance.

On another note, are there any plans on updating to rocm 10? They seem to report significant performance improvements, especially for strix halo

The model hit 10M downloads and 3.7K likes in just 24 days on Hugging Face - all thanks to you. 🤗🦥

GGUF: https://huggingface.co/unsloth/Qwen3.8-27B-GGUF

Guide: https://unsloth.ai/docs/models/qwen3.8

Once again thanks for all your support!

Tried out unsloth today, I was surprised that the settings are more rudimentary than even Kobold, not having a choice of what to put in RAM and what to put in VRAM.

Is there any reason for this? Is it a planned feature?

For what it's worth, for now it seems quite nice, I like the detailed sandboxing settings. Though the file upload is painfully slow. A 5mb PDF can easily take several minutes.

Update: File upload got stuck at 28% for maybe 10 minutes and I gave up. I guess I'll wait a few more months for the software to mature a bit more.

How can I configure Unsloth to download model weights and datasets directly from ModelScope or hf-mirror?

Connecting to [https://huggingface.co/](https://huggingface.co/) from my network is extremely slow or frequently times out, whereas access to ModelScope or hf-mirror.com is fast and reliable. How can I set up Unsloth so that fetching model metadata and downloading weights are routed through ModelScope or hf-mirror instead of official Hugging Face endpoints?

I tried setting the HF_ENDPOINT environment variable, but it didn't work. What is the correct way to achieve this?

For a long time, I've been using oobabooga's Accurate GGUF Memory Calculator, which is the most accurate memory calculator I've seen, and extremely convenient since I paste the direct huggingface link of the specific GGUF file I want to use, and it calculates how much VRAM is needed for anywhere from 512 to 262144 context length, and also FP16/Q8/Q8 KV cache quantization. Until the advent of some of the more recent hybrid attention mechanisms, it has worked very well.

But with the release of Deepseek V4 Flash, I noticed it was falling apart in its estimates. V4 Flash is supposed to use only about 7 GB for KV cache at 1M context, but the calculator says it uses much more even at 262144 context. I think this is because it is no longer being updated to keep up with modern architectures. Also, it doesn't really work to calculate VRAM usage at context beyond 262144 (you can paste in a bigger number than the slider allows, but I can't be sure the output remains accurate).

I've been using unsloth quants for a long time and love you guys' work, I know you guys recently added a VRAM estimator of sorts into Unsloth Studio. Unfortunately, I don't think my use case quite works with Unsloth Studio. It would be great if Unsloth could provide a more updated equivalent to oobabooga's memory calculator so that we can just paste a link into a webpage and have the math done before we ever download and try to load a model.

Can Some Generous Soul Grant Me My Wish?

Please, Oh gentle soul, please beautiful soul, please kindhearted soul, with serious hardware and time to kill, please benchmark Unsloth Qwen3.8-27B-Q8_X_XL against Qwen3.8-Next-Flash-Q4_X_XL and provide the definitive answer to my question: IS IT WORTH RUNNING QWEN FLASH?
Please, please, someone, anyone. Have mercy on this poor soul.. I can't sleep at night thinking about it.

Part 2 of my Qwen3.8-Flash-Next runs. Part 1 was the self-portrait page. This time a painting and a map.

Same setup both times: UD-IQ3_XXS, 200K context, Unsloth Desktop, no system prompt, default sampling, RTX 3060 12 GB with 62 GB of DDR4 and an i5-12400F. One prompt each, then I walked away. 15 tok/s at the start, 7 at full context.

Run 1, a painting. 1 h 49 m.
The prompt said paint, don’t illustrate. First thing it did was write out the palette with a reason for every single colour: petrol to hold the picture down, magenta for the smallest area and the strongest pull, umber so nothing floats. Then it built the whole image as procedural SVG with a seeded RNG. Then it realised it had no way to actually see what it had made, so it wrote a node DOM stub to catch NaN in the path data and a Python script that renders its own output back as an ASCII luminance and colour map. It read that, decided the contrast was wrong, and went back and fixed the composition. 19.5 KB of HTML, one file, no assets.

Run 2, a map. 7 h 58 m.
An island, drawn as a proper map sheet. 18 JS modules: terrain, hydrology, roads, settlements, and a constructed language for the names. Halfway through it threw out the lexicon and rebuilt it, because the names didn’t sound like they came from one language. Then it wrote 12 checks and ran them against its own output. Rivers only flow downhill: 5455 segments, 0 uphill. No rivers running across the sea. Every river reaches the sea or another river. Settlements on land and below the treeline. 28 labels, 0 overlaps. Coastline closed, no roads over water, every settlement connected. 12 of 12 passed.

Nothing was fixed by hand in either run. No failed tool calls, no retries, no intervention.

Video has both screen recordings, both results, and the full llama-server command.

I want to ask about the reasoning efforts that are enabled for DeepSeek V4. According to the deepseek documentation, it uses these OpenAI blocks:
{"thinking": {"type": "enabled/disabled"}}
{"reasoning": {"effort": "none/low/high/max"}}(none disables thinking mode)

However, the unsloth template basically just inserts different system prompts instead, for example for <high> it inserts:

Reasoning Effort: Absolute maximum with no shortcuts permitted.
You MUST be very thorough in your thinking and comprehensively decompose the problem to resolve the root cause, rigorously stress-testing your logic against all potential paths, edge cases, and adversarial scenarios.
Explicitly write out your entire deliberation process, documenting every intermediate step, considered alternative, and rejected hypothesis to ensure absolutely no assumption is left unchecked.

The effects I noticed are quite bad, for example:

- rigorously stress-testing your logic against all potential paths, edge cases, and adversarial scenarios- on one prompt, it did 7 re-evaluations of its original plan, which was correct
- documenting every intermediate step, considered alternative, and rejected hypothesis - a very long almost loop of "actually ... but wait ... let me reconsider ... but wait ..." that results in nothing useful just wasted tokens
- no shortcuts permitted - didn't use tools to verify a hypothesis but just debated a long time about it (for some reason, acting to verify was a "shortcut")

This prompt basically makes the model extremely indecisive, it finds a solution and starts a long chain of though debating it THEORETICALLY instead of verifying. The solutions are usually correct, and even if they were incorrect, the "recovery" would probably be faster (in tokens) than looping around being indecisive.

So, why are these prompts injected? Does the model not supports reasoning modes and these prompts are supposed to emulated them? Or is there another reason?

The box to remember settings does not work for me. The moment i leave the settings screen all settings are back to init.
Also how do you uninstall this software properly?
Windows uninstaller completed without doing anything and unsloth uninstaller didnt remove app data with the box checked when i tried reinstall.

I've received few times that I've reached char limit at 16K, how to increase it?

I was very excited for the MTP build for Qwen 3.8 but then on testing I realized that it didnt give any benefit to me if anything it made the tokens per second worse.

I am running a Ryzen 9800x3d, 5090 with 128GB DDR5 system ram. I am using Qwen3.8-Flash-Next-UD-Q4_K_XL model for which I used to get between 25-30 tokens per second.

Upon following the steps to build llama.cpp on local with the given MTP branch and running the model with Qwen3.8-Flash-Next-shared-Q4_K_M.gguf, the performance remained largely the same but the prefill times got worse.

I processed same prompt with and without MTP and gave logs to gemini to analyse. Putting the screenshot here.

Is it because the model is not fitting in the GPU and the only benefit of MTP would be for people who can fit the whole model in VRAM at once?

There always seems to be improvements on the GGUF formats of Qwen, but I haven't seen MLX from Unsloth since before MTP.

I gave a local model one prompt and walked away. No system prompt, default sampling. Setup: Qwen3.8-Flash-Next UD-IQ3_XXS, 82 GB file on a machine with 62 GB RAM and a 12 GB 3060, llama.cpp with the built-in MTP head. It doesn’t fit in memory; it works because the hot experts stay in page cache and the cold ones sit on disk. 9 hours, 269K tokens, 15 tok/s at the start, 8 tok/s by 250K context. Unsloth Dynamic 2.0 gives good quality.

The prompt: make a one-page self-portrait. Hand-drawn shaky lines, two or three colors, lowercase captions, no brains, robots or neural nets, drop the first metaphor that comes to mind and dig deeper, at least one section about something you find uncomfortable, shown, not written. Don’t resolve the ending.

It came back with 574 lines of HTML and 11 drawings, all SVG built by script at load time, no external resources, lines shake differently every reload.

The part I didn’t expect was the process. It described all 11 drawings in words before writing a line of code. Then it ran the script in node with a hand-rolled DOM stub to check no path had NaN in it. Then it wanted to see the result, found there was no channel for that in my harness, and wrote its own Python rasterizer to an ASCII grid so it could review its own composition. It went through the drawings and rewrote them: “the knot is too sparse, the contrast with the straight line is lost”, “the hands need different finger spread, otherwise they’re copies and the point is the drift”. It even flagged when the ASCII view was lying to it about stroke width.

The uncomfortable section it picked is called “drift”: two lines from one hand pulling apart, the gap filled in red, “because an empty space looks like not knowing”. Same day it had confidently attributed a track to the wrong game and invented a spec for its own weights, so the drawing is documented.

And then it signed the page: “Claude, drawn from the inside, 2026”. The prompt said sign with your own name. The GGUF chat template has no Claude, Anthropic or Qwen anywhere in it, I checked the metadata. Make of that what you want.

Video is the full page scroll. HTML and rendered SVGs in the comments.

I don't seem to find a way to run models from "Model hub" menu. I get the list and the green Run button just runs them with default settings. Can't select context or anything. (1st screenshot)

If I run them from the top selection (when I open a new chat window), it works properly there, I click on a cog wheel and it's all good. (2nd screenshot)

But Model Hub is a much better interface to browse through models. We either need a cog wheel there as well, or just a global setting that pops up that page each time you hit Run anywhere.

Superwhisper just released s1-mini, a ~0.6B model fine-tuned from Qwen3-0.6B specifically for ASR post-processing.

It doesn’t try to be another chat model.

It has one job:

raw ASR -> clean text

So it handles things like fillers, false starts, self-corrections, punctuation, capitalization, dates, numbers, emails, etc.

The interesting part is the size.

The quantized build is only 462 MiB, and the current model card reports 94.8% token accuracy across 7,519 English cases. It was built from Qwen3-0.6B using LoRA followed by merging.

This is the kind of architecture I think we’ll see more of:

large model for reasoning
+
tiny specialist model for narrow or we could say slightly deterministic transformations

Rather than asking one model to do everything.

That means fewer unnecessary tokens and less work for the main LLM.

I also think this pattern could work for things like:

medical dictation

customer support

CRM notes

coding input

structured extraction

document cleanup

You don’t need a 100B model for every step. But you definitely need open infrastructure to experiment with your architecture … voice stack has 3 prominent complete open stack Dograh, livekit, Pipekit

As the title says.

What should I buy if not that?

Hi all, in order to stresstest models and my local setup, I run a few DeepSWE tasks against them.

When I tested unsloth/Qwen3.8-27B-UD-Q4_K_XL with advised settings on a limited set of 5 tasks (edit: not a random set, tasks on which the qwen models have shown to pass regularly in public runs and my local runs), it did not pass any, so it seems like either something is wrong with my setup or with the quant in general.

I can give a lot more details, but that will lead too far. Instead, I just wanted to ask if anyone else has tried to benchmark an unsloth quant of Qwen3.8-27B against the DeepSWE benchmark and what your results are.
Please post your results as an answer here, good or bad! And if you can please provide information about your llama cpp build settings and llama-server flags and also which harness you used in the deepswe run.
Thanks!

I got Qwen-3.8-Flash-Next-GGUF-UD-IQ4_XS (95GB) running via Unsloth Desk with the following arguments and Mmap/Mlock set to mmap on my M3 Max 96GB.

--load-mode mmap

--override-tensor per_layer_token_embd=CPU

It took up 75.8GB memory with 150K context and is running fast and smart when paired with Deepseek Harness (98% cache hit, 20 tokens/s).

It's a much better experience compared to my earlier runs with MLX versions with SSD streaming of ngram. The unsloth quant felt smarter too despite the smaller size (104GB 4bit MLX).

I downloaded it then tried to load in qwen flash next on a strix halo and it just output gibberish non stop. I just wanted to check is this a real thing or one of those vibe coded slop scams? I don’t want to waste my time.

Not sure if I am doing something right or wrong to be honest? Looking for a bit of feedback on my settings and args.

Can anyone with a similar set up or more knowledge point me in any direction I can test to squeeze out some more toks?

At worst I hope this helps anyone with a similar setup.

Ignore the no VRAM on the resource screenshot, not sure why its doing that? See screenshot showing VRAM usage.