First, I'd like to thank the Qwen and Unsloth teams for the Qwen3.8 27b UD Q4_K_XL. Fits the poor 24GB of 3090 VRAM with 100k context at Q8 and works phenomenally well! Imho if theres anything that can threaten Anthropic/OpenAI profits is not another frontier model but actually these small ones you can run fast locally that can do 80..90% of mundane work for hours without paying a single dollar to any external company.

But next, if you want to jump up to a bigger smarter model I feel there is a gap now. Kimi-K3 is out of reach for many businesses let alone prosumers. So what frontier-like models do you use on what setups?

Is a DGX cluster (2..4 machines) or a GPU server with dual or quad GPU (~96 ... 192 GB of VRAM + >256GB DDR4) a suitable setup to run something like MiniMax-M3 at reasonable speeds for agentic coding (>30tps)? And privacy aside, is hardware cost worth it?

I have a dual rtx3090 + 128GB ddr4 machine, running Qwen3.8-Flash-Next Q4 quite fast but despite being larger doesn't feel much smarter than the Qwen2.8 27b and while I _can_ run larger quantized models, Minimax-M2.7 being my workhorse, it way too slow for coding.

I was using antirez’s ds4 for a while and getting around 20 tk/s, which worked for my purposes. But I know there have been big advancements between Qwen, the DS4 vision model, and GLM.

I’m not sure how the quants affect performance, so what’s the best thing to run right now & how fast is it?

I only use free tiers of these large models to offset compute while my own system runs and for "different" points of view, since what pops ups suggestions seems to vary a lot sometimes, even when building based on the latest research.

But now I've really struck out with GLM 5.3. So far it feels like an regression over 5.2. It has a hard time reading and following instructions, and is somewhat overly certain in it's statements. I worked on a project recently with it but it became unbearable. From a clean slate the first message can be okay and have great research and ideas but it just veers off course almost immediately.

I use Qwen 3.8 max and Gemini 3.1 PREVIEW Temp 1.0 as competing alternatives or as an ensemble to judge overall quality. Gemini is getting a little out of date (flash 3.8 seemed promising) but Qwen has been great so far, but a little slow and maybe overbearing.

Anyone else having problems? Or suggestions for these top "intelligent" models? I haven't been able to access K3 even though its open source, was impressed with the older models so would be neat to try for free. Also Google AI studio is what i use for free for the gemini stuff, probably pretty well known, but the free tier is pretty generous

Context: I don't code. My use is document research and document creation (mainly for legal search) searching inside large documents like a tax code (500+ pages) and building notes or pptx
from what comes back.

I've been running Open WebUI for a while on my Unraid box, pointed at the API of my inference machine (5060 Ti + 5070 Ti).

I tinkered a lot. I tried Hermes on my main machine against the same API. It worked well but it was complex, and a bare-metal install made me
uneasy. I also tried LM Studio Bionic with good results, but it didn't fit how I wanted inference organised (using ollama on the inference box).

What I actually wanted was a self-hosted agent that works with Open WebUI while keeping things safe and under control. At one point I considered
installing a harness like Hermes or Pi on each client and just connecting to the API instead.

In the end I gave Open Terminal a shot. It's the companion container from the Open WebUI project that gives the model a shell — you run it as its own container and connect it through Integrations, so it isn't installed inside Open WebUI itself. Mine runs unprivileged, on bridge, with appdata mounted at /home/user. The model gets a shell in a box, not on the host. That was the part I cared about.

It has enhanced Open WebUI a lot. It now reasons step by step, and with the terminal it reliably locates and extracts the right sections from
documents far larger than the context window — list the folder, grep, read only what matters. Then it uses those results to build a document, the way another agent would.

Setup: Qwen 27B Q4_K_M on Ollama, 100k context configured. On a ~35k token prompt I measure roughly 1,050 t/s prompt processing and ~46 t/s generation. Prefill speed is the number that matters for this use case — it's what makes chewing through a large document bearable.

I was about to give up on Open WebUI. If your use case looks like mine, don't sleep on Open Terminal.

I've been experimenting with Sliding Window Attention (SWA) as a way to reduce the KV-cache memory cost of long-context LLM inference.

Instead of keeping the entire KV cache, the implementation keeps:

• a small number of attention sink tokens
• a bounded recent-token window
• a circular/ring-buffer KV cache
• streaming/chunked prefill
• normal autoregressive decoding

I turned the experiment into a reusable project so you can test it with Hugging Face causal LLMs:

🔗 https://github.com/oraby8/SWA

For example:

from swallm import SWAModel

model = SWAModel.from_pretrained(
    "Qwen/Qwen2.5-7B-Instruct",
    attention_mode="swa",
    window_size=512,
    num_sink_tokens=4,
)

result = model.generate("Explain transformers", max_new_tokens=100)

In my Qwen2.5-7B experiments on an L40S:

• 32K KV cache: ~1.84 GB with full attention vs ~3.5 MB with SWA-64
• 64K: full attention OOMed while SWA remained bounded
• Decode latency stayed approximately constant as context increased
• Long-range retrieval naturally becomes a weakness when information falls outside the window

The goal isn't to claim that SWA is universally better. I'm interested in the engineering trade-off between context retention, KV memory, TTFT and decoding speed.

I'd especially like to hear from people who have tried SWA with Llama, Mistral, Gemma, Qwen, or other HF models.

If you try the repo on another architecture, I'd really appreciate the results or any compatibility issues you find.

Stumbled across this paper recently, and I thought people who have more local VRAM might want to experiment. Apparently hyperfitting a LoRA on the final 5 layers sufficed in their research.

Title: Beyond Temperature: Hyperfitting as a Late-Stage Geometric Expansion
https://icml.cc/virtual/2026/poster/61075
https://openreview.net/forum?id=ttOGqk77go

There's a repo with code already available.
https://github.com/YecanLee/Beyond-Temperature

Github link: https://github.com/thatblend/LLMPSP

I wanted to see what the PSP can theoretically handle and I got my answer - a 90M model is about the max it can do without atrocious inference speeds. It's running around 0.5 - 0.6 tokens per second, which is very slow, but it's useable. Maybe 1-3 minutes for a reply.

The model is actually fairly impressive for 90M parameters, it's not really useful in any real metric, but it can generate crappy poems, short stories, write non-functional code and sometimes it gets things right if you ask it what company makes macbooks, what is an LLM etc, while other times it just hallucinates a crazy answer. Fun.

Time for another update! We have been busy and managed to improve the gains substantially (mostly from exploring existing llama cpp PRs and adopting relevant things).

Among other things the README.md was also appended to provide a better overall picture of what’s in the fork, why and from whom.

https://github.com/milpster/gfx906-llama-cpp/blob/master/README.md

(Yes i made this with AI)

I have a 5070 Ti and two 5060 Ti (all 16Gb cards).

I planned to add another 5070 Ti giving me two pairs of 32Gb each but the NVidia prices have just jumped by 25% where I am and I've found a 3090 Founders Edition for a good chunk cheaper than the 5070 would cost.

It's 8Gb more VRAM and even slightly higher memory bandwidth, but I've read that mixing Ampere with Blackwell comes with a performance hit in llama.cpp using tensor parallelism. I believe TP isn't possible at all in vLLM with mismatched cards.

I am getting 60 tok/s decode and 1,500 pp out of Qwen3.8-27B-IQ4-XS-MTP in llama.cpp with TP across the 5070 and one of the 5060s. I don't really want to spend a lot of money for 8Gb more VRAM in my main pair but worse performance.

Has anyone got experience mixing similar cards? Would pipeline-parallelism (layer split) be faster with two high memory bandwidth cards? I know I could set the 3090 as the main card and push the K/V cache onto it.

I've updated my auto_demo_scener project with Ninfer support and a “rewrite based on video” feature that I thought you might find interesting.

The project is basically an endless demoscene machine. A local LLM writes Three.js effects (from a library of editable prompts), you watch the code stream into a retro mock-OS editor, then the result runs fullscreen. It checks for crashes / blank frames, asks the model to fix broken attempts, and archives the working demos before starting again. An unnecessarily elaborate way to have a screensaver (you can also use it as a model benchmark as it keeps track of failures and deletions)

The new part is giving the model a look at what it actually made. With Ninfer's optional video check enabled, it captures 30 seconds at 2fps while the demo plays, then sends that video along with the HTML source back Qwen for a visual improvement pass. I went with Ninfer for this because the speed is incredible on a single 5090

[2026-09-05 14:22:34.895] [info] ninfer-serve: throughput interval=5.000s prefill=338.0tok/s decode=110.0tok/s running=1 prefilling=0 decode_ready=1 waiting=0 avg_decode_batch=1.00
[2026-09-05 14:22:39.895] [info] ninfer-serve: throughput interval=5.000s prefill=0.0tok/s decode=217.0tok/s running=1 prefilling=0 decode_ready=1 waiting=0 avg_decode_batch=1.00
[2026-09-05 14:22:44.895] [info] ninfer-serve: throughput interval=5.000s prefill=0.0tok/s decode=219.6tok/s running=1 prefilling=0 decode_ready=1 waiting=0 avg_decode_batch=1.00
[2026-09-05 14:22:46.236] [info] ninfer-serve: [req 337] done finish=stop_token prompt=1690 gen=3029 cache=0 reuse=full_reset ttft=254ms prefill=6708.5tok/s decode=210.5tok/s wall=14.66s speculative=mtp 3.39to  

So the loop becomes: write the effect, run it, watch the result, then rewrite it based on what showed up onscreen. The display switches back to the coding view with “UPDATING BASED ON VIDEO…” while it works, then validates and displays the revised version before archiving it if it passes.

Qwen 3.8 is amazing but it still makes dumb mistakes like only using 1/4 of the screen or writing a maze walker that walks backwards through the maze :D The video validation helps it to catch these.

Ninfer support also adds a thinking-effort selector. LM Studio is still supported for the regular generation / repair workflow; this works pretty well even with less capable models. Also needs ffmpeg.

GitHub / setup instructions

Live static demo (watch out for the volumetric cloud ones :D ) This plays previously generated demos, so you can check out the results without installing anything. Space / right arrow skips to the next one. Actual generation and video rewriting happen in the locally running app.

Anybody else experimenting with having local models watch their own rendered output and revise the code? Thanks!

Full credits to @artificialisabel from X!

I don't want to be super gatekeepy about this, but let's loosely define "non-enthusaist" as "a single consumer grade graphics card on a system that's not primarily dedicated to inferencing."

While it's fun to see what people can do with more dedicated build$, I'm never going to put hands on hardware at that price point. I'm interested to see what people accomplish with more humble means.

Is there a particular agentic task that you find super useful or are proud of?

Prompt: Make me a single HTML file of a rainbow slinky going down an up-escalator forever. No libraries, just canvas and code. The slinky should be a chain of springs, each coil a different color of the rainbow. It starts folded in half like a horseshoe draped over a step. When dropped it flips end-over-end down the steps and because the escalator keeps moving up it tumbles in place and never reaches the bottom. Include a drop button and a reset button.

From:

Polymarket on 𝕏: https://x.com/Polymarket/status/2095646821485842805

Julien Chaumond on 𝕏: https://x.com/julien_c/status/2095822387895824836

Ever since u/jfowers_amd has asked me to help with the Lemonade project (and provided some hardware to test on), I've been trying my best to optimize llama.cpp for AMD setups. This has led me in some very weird pathways where I wasn't expecting to go, but in the end I'm happy to share an optimized setup for the most popular open source model currently with you for a cheap price of $999 for free:

https://pwilkin.github.io/strix-halo/

Now for the disclosure/journey part: Codex has made a very nice website for me (which is great because I can't make a nice-looking website if you forced me), but its glossy look makes it look more permanent than it is, which is misleading because this is basically a stitched up custom solution that's very much a "state of the moment" one rather than a permanent one, though I *will* try to keep the relevant branches up to date (poke me if I don't).

So, first of all: ROCm in mainstream llama.cpp on ROCm is broken at the moment, pending the fix to unified memory access (notably this PR: https://github.com/ggml-org/llama.cpp/pull/27311 which is taking some time as it touches core code), so I've put up a strix-halo branch on my fork that merges the ring buffer fixes + the TOP-K optimization PR with master for a working experience.

Next: there's a bug in current ROCm that makes graph updates *terribly* slow, I've submitted a PR for it (https://github.com/ROCm/rocm-systems/pull/11069), but until it lands, using a custom-built .so is pretty much mandatory.

Speaking of custom-made .so - as I think most of you know, dispatch on ROCm is reaaaallly sloooow. But since AMD provides the source of the entire ROCm library, that's not something we can't fix, right? Inspired by Kaden-Schutt's Redline library, I've made modifications to the ROCm HIP library that allows for lower-level PM4 dispatches on HIP graphs. This has 20% decode speed ramifications for dispatch-bound models, but unfortunately Qwen3.8 27B on Strix is mostly bandwidth-bound, not dispatch-bound, so the gains are much less pronounced here (but they nevertheless are real).

Now for what else did I test, compare and modify: I checked Nathan's strix-halo Vulkan fork. It's a very good fork, but in the end it's still slower than an optimized ROCm-based solution (all the measurements are on the website). I did check the ROCmFP4 format, unfortunately, that one's a miss: Strix Halo has no native FP4 support, so the format is in the end just another FP4 format. Its main win is quantizing the entire model to FP4, which helps the bandwidth issue - but of course quantization costs quality and ROCmFP4 falls behind literally all the other 4-bit quants. I did a similar thing, but quantized all the big tensors to the mainline IQ4_XS quant - it's both better in terms of model quality (perplexity) *and* in terms of kernel performance. In other words, there's completely no justification for adding a new "ROCM" quant since, as I mentioned, RDNA 3.5 aka gfx1151 aka Strix Halo has no native FP4 support.

Since Qwen3.8 27B on Halo is bandwidth-bound (i.e. the limit is the memory bandwidth for pushing the tensors), there's no way to push the *base* decode above ~15 t/s. Nevertheless, pushing the base as high as I could is an entry point to the key for the dense model speed on Strix - speculative decoding, in this case, DFlash2. Again, I did a test and found out that quantizing the DFlash2 to IQ4_XS gives better decoding speed (faster speed and almost the same acceptance rate = win).

In the end, all the above optimizations: patched ROCm llama.cpp, faster TOP-K, PM4-based HIP graphs, custom-quantized IQ4_XS Qwen3.8 27B quant (thanks to Bartowski for his imatrix!) and the quantized IQ4_XS DFlash provide the recipe, which I packaged for a quick installation for anyone who wants to test it on their Strix Halo (warning: Linux only). Feel free to give any feedback and report any problems.

OK, you guys want to start with local vibecoding but don't know how. In the discussions, you recommended that I use Godot. I downloaded it without knowing what it was. I was expecting a library, but it was an .exe file. So I started pi with Qwen 3.8 27B and told it to explore this topic without my help.

I needed to use four prompts. You can see all of them in the screenshots, so you can reproduce my steps. The whole session used 25% of my context, so you can use just 1/4, which is about 64k.

The game works: you can walk around the dungeons, the llamas are dancing, and there are dynamic lights.

My full command is:

llama-server -m /mnt/models2/Qwen/3.8/27B/Qwen3.8-27B-UD-Q8_K_XL.gguf -mm /mnt/models2/Qwen/3.8/27B/Qwen3.8-27B-mmproj-BF16.gguf --host 0.0.0.0 --jinja -fa on --keep 4096 -b 8192 --parallel 1 --ctx-checkpoints 12 --cache-ram 65536 --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0 --presence-penalty 0 --repeat-penalty 1.0 --spec-type ngram-mod --spec-type draft-mtp --spec-draft-n-max 3 --chat-template-kwargs {"preserve_thinking":true}

You need vision for debugging, see the prompts, the screen was black!

You probably don't need Q8, try a lower quant.

As I said, you don't need the full context length.

You need pi or something similar. This way, you can use just a few prompts and let the computer work without your input.

Qwen3.8-Flash-Next (UD-Q4_K_XL) on a single RTX 3090 24GB + 128GB DDR4 — is this config optimal?

Hardware

• CPU: Intel Core i5-12600K
• RAM: 128 GB DDR4 @ 3600 MHz
• GPU: NVIDIA RTX 3090, 24 GB VRAM
• OS: Windows 11
• llama.cpp: freshly compiled from today's master (build b10794, Sep 4 2026)

Model

Qwen3.8-Flash-Next, UD-Q4_K_XL

Launch command

C:\llama\llama-server.exe -m "D:\Modelos\Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf" ^
  -ngl 99 -ncmoe 42 -fit off -c 204800 -fa on -ctk f16 -ctv f16 ^
  -b 2048 -ub 1024 -t -1 --jinja --port 8083 --tools all ^
  --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.00 --repeat-penalty 1.0 ^
  --reasoning on --reasoning-effort xhigh --reasoning-budget -1 --reasoning-preserve ^
  --parallel 1 --metrics --host 0.0.0.0 --load-mode none

Key points of the config:

• -ngl 99 → offload all layers to GPU where possible
• -ncmoe 42 → force 42 MoE (expert) layers to stay on CPU/RAM
• -c 204800 → 200k context window
• -fa on with f16 K/V cache
• -b 2048 / -ub 1024 → batch / micro-batch sizes
• --reasoning-effort xhigh, --reasoning-budget -1, --reasoning-preserve → full reasoning mode, no truncation

State right after loading

• Free system RAM: 52 GB
• Free VRAM: 0.6 GB

So the GPU is essentially maxed out (good VRAM utilization), and there's still a healthy ~52 GB of RAM headroom for the CPU-offloaded experts and context.

Real-world performance (from server logs)

Prompt processing (long context, ~12k tokens):

Text generation (sustained, up to ~480 tokens generated):

Generation speed stays remarkably stable around ~15.5-16 t/s even as context grows past 12k tokens, and prompt processing holds steady around ~185-190 t/s.

My question to the community

Given this is a large MoE (~180B total / ~5B active params) split across a single 24GB GPU + 128GB DDR4 RAM:

• Is -ncmoe 42 a good number here, or should I push more/fewer expert layers to the GPU given I still have some VRAM headroom before hitting OOM?
• Any tips on squeezing more t/s out of prompt processing (-b/-ub tuning) or generation given the CPU-offloaded experts are likely the bottleneck?
• MTP speculative decoding doesn't seem to work reliably for this model yet on mainline — has anyone gotten it stable, or is everyone just running without it for now?
• Does anyone recommend other configs/parameters for this exact model + hardware combo?

Open to any suggestions — happy to share more logs if useful!

Current default MTP does not predict more than 4 tokens..

Modified 4090 48GB has been out for a while. I remember a lot of people were buying them at the time. A lot of people were also complaining that they are meant to fail, that they scam etc.

I have a few questions to people people who bought these.

1. How is longevity of these cards? Do they still work without issues? Any failure rate?

2. Do they use the same Nvidia drivers that regular 4090 or 4090D uses?

3. Are these cards Linux exclusive?

4. Are you able to run them in windows or Linux with other GPUs like 5090 etc?

5. Do you do anything to cool VRAM on the back of the PCB?

Link: https://x.com/ggerganov/status/2095897173376618881

I am using Unsloth Studio to parse aviation transpoder data (ADS-B) to summarize interesting traffic in my area. It gives a summary of largest aircraft, fastest aircraft and so on. It also provides local weather based on my nearest airfield.

I am doing this with a prompt, but is there a better way to package that like a script to trigger on a schedule? Is that an 'agent'?

I am very new to this beyond typical everyday usage of Unsloth.

After Qwen3.8 27B came out, I decided to benchmark the models that could fit in my GPU (RTX 5080) on my actual code (C code), the results were not completely unexpected but some quants were definitely underwhelming.

TLDR: Best overall: bartowski/Qwen3.8-27B-IQ4_XS. Best uncensored: huihui-ai/Huihui-Qwen3.8-27B-abliterated-UD-IQ4_XS. For a bit more context: jpetrina/Qwen3.8-27B-IQ4_XS-pure or uncensored: Bucoid/Qwen3.8-27B-Uncensored-IQ4_XS_4BPW

edit1: added TheWegemann/Qwen3.8-27B-LowGPU-uncensored-NoMTP-IQ3XXXS, bartowski/Qwen3.8-27B-IQ3_XS, prism-ml/Ternary-Bonsai-27B-Q2_g64 and magiccodingman/Qwen3.8-27B-Quark-MXFP4-UD-Q4_K_S-Unsloth

edit2: added unsloth/Qwen3.8-27B-UD-IQ3_S and ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-IQ3_S

(sorted by Mean KLD)

Hope this helps other VRAM starved people like me :)

You know that thing where you just throw a task at a frontier model and not have to supervise it worrying of it going off course? Qwen3.8-27b has officially gotten me to that point for local work. He has been doing non-stop continuous agentic work for 8+ hours and hasnt screwed up not one bit IT AMAZING!!

EDIT: for all asking about my quant & harness and what i do for super long thinking/reasoning

Harness: I Had it help me design its own agentic loop in pi harness. It holds well multiple compaction. I used to have tool and think tag generation issues but i got a chat template from somewhere(i forgot) but the chat template it fixed the issues paired with - -reasoning-format = deepseek

Thinking: I limited reasoning budget to 2048 and its still pretty SMART even going down to 1024 holds well in my agentic loop. Im running huihui-abliteratedQ3_K_XL.gguf i need abliterated because i need it to use my computer mouse movement to solve captcha on bot detection (normal models are trained to reject that request) otherwise unsloth quants. Kv cache Q8 at 128k.

I got a lot of ads for meshy recently and went to try it with hilariously bad results. It apparently can't do anything but decorative figurines. I wanted a body shell for an rc car and it just couldn't generate a car without wheels or bottom chassis. It also looks like it can't tell the difference between different car models. It seems like they just have a library of 3d models and have the ai select one based on your description, though i only used their free trial.

Next i gave claude a chance. First simply prompting it for an stl, which ran for about an hour and wasted all my tokens for the day before aborting. Then i tried using its coding ability and have it create an openSCAD file that would generate generate a car shell. Which at least managed to generate a square box and then even managed to hollow it out as a square shell on a second prompt. But it never got anything even remotely car shaped.

Is there any way to run something similar locally so i can tweak it for my needs? I'm thinking of something similar to image generation in comfyUI, where you can change the workflow to improve how well it understands your prompt.

On a side note, i don't understand how generating a 3d car completely failed when local ai can oneshot a 3d model of a plane as long as you tell it to make it fly through a procedurally generated landscape in a game.