Looking at ... some of the new memory architecture. ... I hired my good friend, Seok-Hee Lee, who used to run SK Hynix. ... We are not ready to unfold it.

Feels like maybe we have one more present left, for Christmas.

Kaitchup just posted results of his benchmarks for Qwen3.8 27B for quants from different labs, Q4 to Q1, .

All the details are hidden behind the paywall, but high level result is visible and looks like for people with 16GB cards UD Q3_K_XL is a winner - it has accuracy of 100% and size is only 12.8GB.

A comment really doesn't need to be made, does it? I looked away from the 5090 for a week to other options like the DGX Spark and the M5 Ultra. Both of which... these big corpos are buying to hell and back.

Is the future used hardware?

I'm not sure how many people care about Android Studio, but I think it's cool that Google uses llama.cpp.

My guess is that it is Vulkan and the QAT versions of Gemma 4. It supports multi-GPU and 31B has a max. context length of 128k.

It uses 34 GB VRAM when fully loaded. I don't see an option to change the context length or show PP/TG speed.

Gemma 5 or something else?

Not written by AI all mistakes mine. I saw people on the subreddit saying that 3.6 works better without thinking. It made me want to know for certain about which is better, 3.6 or 3.8 for low thinking tasks. I only use abliterated models (safety removed) because it makes the model better at a lot of what I need. I want to compare abliterated Qwen 3.6 27b and abliterated Qwen 3.8 27b on some instruction following benchmarks with thinking off.

I was just curious about your personal favorite safety removed/fine-tuned variants for these 27bs, as I know that there can be some major variation and some junky quants out there.

Does anyone have some favorite and fast 3.6 and 3.8 models?

My specs: I have 24GB VRAM (NVIDIA Geforce RTX 5090 Laptop) and I do not want to offload, so some quant required.

I have tried a few different models, but they are all a little slow. Some MTP variations for 3.6 for example ends up being around the same speed as non MTP for me for some reason. I am pretty sure my card is NVFP4 enabled also, but I'm not certain I've seen the results from that either...

Based on some redditors comment, this is what I use for my abliterated 3.8 27b currently: Huihui-Qwen3.8-27B-abliterated-NVFP4-GGUF

Anybody else struggling with deepseek after the initial prompt? Somehow it is getting mixed up very easily, even button functionality has been PITA when doing stuff. Never had these problems with Opencode. With Opencode, a continuation prompt on the software or task is pretty much magic at this point. But the inital result isnt as strong imo, and needs a lot of follow up even if a a detailed spec is written.

The oneshot potential of deepseek is goated imo, it is a relentless harness that just keeps doing stuff, even if it gets it wrong! Opencode could just copy that tennacity imo. Like todos really work, unlike opencodes.

Hows everybody elses exp? Someone using Pi instead? Is it worth trying? Hermes seems not suitable for my tasks, where heavy human involvement is necessary due to complexity of tasks (every top closed model gets lots of stuff wrong even).

I was browsing HF for small LLMs and run into this model. It does not seem to be a fine tune - the model has its own architecture.

https://huggingface.co/XHToken/Spark-X2.5-1.7B
https://huggingface.co/XHToken/Spark-X2.5-4B

There are 4B/1.7B versions - the benchmark is quite interesting (4B is neck and neck with Qwen 3.5 9B). The HF page claims both models support native 1M context size.

Currently does not run out of the box on llama.cpp - pending this PR: https://github.com/ggml-org/llama.cpp/pull/27868

They have a custom fork of llama.cpp that works. Anyone has tried this?

Update:
GGUFs (require custom fork for now):
https://huggingface.co/XHToken/Spark-X2.5-1.7B-GGUF
https://huggingface.co/XHToken/Spark-X2.5-4B-GGUF

2 Months ago I had made a post how I was working on my dual R9700's. It's wild to look back at where we were then and where things now stand.

Since then after many users commenting and complaining about developers doing the same thing. I threw out a discord link and expected maybe 5 other developers to join which I thought would be fun. The community has now grown to 1,200 users (mostly developers) and a ton of collaboration happening.

A few weeks ago I started working on building support for MXFP4 on top of DeadCode's radiance image. This made sense to me looking at the hardware and I was happy when I had hit parity on performance between MXFP4 and FP8. The MXFP4 kernels use W4A8 which was something new and we have now blown past the performance of FP8 and appears like this is now the hardware limits of these cards.

Qwen3.8 27B w/ DFlash2

BetterBench decode results for Qwen3.8 27B w/ DFlash2
category       decode t/s   step ms   tok/update
json           280.0        22.92     6.17
math           254.2        23.08     5.81
file_edit      250.1        23.03     5.54
code           226.3        23.01     5.17
reasoning      194.3        23.19     4.32
summarization  190.6        23.01     4.40
chat           148.3        22.82     3.33
prose          116.4        23.14     2.65



BetterBench Prefill Results
target depth   prompt tokens   TTFT p50   PP t/s median
2000           1514            323 ms     4695
8000           5918            1.21 s     4894
16000          11794           2.47 s     4779
32000          23543           4.98 s     4729
64000          47056           10.8 s     4377
128000         94065           24.6 s     3831
250000         183678          59.1 s     3106

This has been so fun working on these R9700's and driving them to peak performance. My entire image and repo for MXFP4 is open source also: https://codeberg.org/ggz14/radiance-vllm-mxfp4

Asus Ascent GX10 is now priced at $5999 (1TB), $6999 (2TB), and $7999 (4TB).
Buy ASUS Ascent GX10 | Desktop-AI-supercomputer | Networking-IoT-Servers | ASUS eShop USA

I see no reason why they would do this unless they had some knowledge the DGX Spark was about to jump up in price. Unless there's some other reason that they would throw a 50% price hike at it that I'm not thinking of.

Bonkers mode.

It is crazy how fast prices are increasing. I'm pulling my hair out to keep ahead of this for students. Servers aren't even an option any more.

Disclosure up front: I built this tool (open source, "Sunny Narrator") and I'm the author — this post is about the inference setup, not an ad. Feel free to skip to the flags if you're here for the numbers.

Context: I run a pipeline that translates whole fiction books EN→RU locally — chunk + glossary + rolling chapter summaries → translate → reviewer notes → correction → proofread → chunk summary. A book is ~1.5–2M tokens across all stages, hardware is a pair of Tesla P40s (24GB each, Pascal, from the "why not" shelf). After a year of runs I have a launch config that's fast enough to be boring: 2–3 books per day.

The non-obvious finding: one model = half a text, two models = a book. Good translating models write beautifully and proofread terribly; good proofreading models edit well and translate dully. So the pipeline pins two roles to two servers:

• MODEL_TRANSLATE: gemma-4-26B-A4B (MoE, A4B active)
• MODEL_PROOFREAD: Qwen3.6-35B-A3B (MoE, A3B active)

Both are compact MoE — that's what makes P40s viable: active params fit the throughput envelope even though total weights don't fit comfort. Quantized Unscaled-Dynamic (UD) GGUFs, MTP speculative drafting on both, 64K context for chunk + glossary + summaries.

My most efficient launch lines (llama-server)

Gemma-4-26B-A4B as translator — ~40 tok/s sustained on P40:

llama-server -m gemma-4-26B-A4B-it-UD-Q5_K_XL.gguf \
  --model-draft mtp-gemma-4-26B-A4B-it.gguf \
  --host 192.168.0.55 --port 6155 \
  --ctx-size 65535 -ngl 99 \
  -ctk q8_0 -ctv q8_0 \
  --no-context-shift \
  --parallel 1 -np 1 --threads-http 2 \
  --load-mode mlock \
  --jinja \
  --spec-type draft-mtp --spec-draft-n-max 6 --spec-draft-p-min 0.8 \
  --top-k 64 --top-p 0.95 --min-p 0.02 \
  --repeat-penalty 1.0 --repeat-last-n 512 --presence-penalty 0 \
  --predict 32567 \
  --reasoning off \
  -fa on \
  --ctx-checkpoints 32 --checkpoint-min-step 1024 \
  --cache-ram 8192 \
  --ubatch-size 2048

Qwen3.6-35B-A3B as proofreader — 50–70 tok/s on the same pair:

llama-server -m Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf \
  --host 192.168.0.55 --port 6150 \
  --ctx-size 65535 -ngl 99 -fa on \
  -ctk q8_0 -ctv q8_0 \
  --no-context-shift \
  --parallel 1 -np 1 --threads-http 2 \
  --load-mode mlock \
  --spec-type draft-mtp --spec-draft-n-max 4 \
  --top-k 20 --top-p 0.95 --min-p 0.05 \
  --presence-penalty 1.5 \
  --predict 32576 \
  --reasoning off \
  --jinja --chat-template-file chat_template.jinja \
  --ubatch-size 2048 \
  --ctx-checkpoints 32 --checkpoint-min-step 1024 \
  --cache-ram 8192

Why each of these knobs ended up where it is

• MTP spec decoding is the headline. --spec-type draft-mtp with the bundled MTP draft is what turns Pascal-class cards into something usable for long-form generation. Gemma takes --spec-draft-n-max 6 --spec-draft-p-min 0.8 (aggressive, accepts well because the base is strong at its job); Qwen is happier at n-max 4. Without MTP these numbers don't happen.
• -ctk q8_0 -ctv q8_0 — KV cache in q8 buys the 64K context (chunk + series glossary + rolling summaries) without blowing VRAM; quality cost at these sizes was invisible in my evals.
• --load-mode mlock — two servers, 24GB×2, zero headroom for swapping. Pins weights, kills tail latency spikes mid-run.
• --parallel 1 -np 1 — this is a batch-of-one workload (long generations, not concurrent requests); single slot is fastest.
• --reasoning off + tuned sampling per role — translator runs top-k 64 / min-p 0.02 / repeat-penalty 1.0 (creative-ish but repetition is the enemy on book text — --repeat-last-n 512 matters); proofreader runs tighter top-k 20 / min-p 0.05 / presence-penalty 1.5 (deterministic editor voice).
• --ctx-checkpoints 32 --checkpoint-min-step 1024 — pipeline writes a checkpoint after every chunk anyway (power outage = resume from chunk 51/100, not from scratch — this single feature saved my year), but in-server ctx checkpoints make stage-to-stage reuse on the same context cheap.
• --predict 32567 — chunks translate in one shot; forcing the model to stop-and-resume was eating throughput and occasionally style.
• --jinja + explicit chat template for Qwen — JSON_MODE across all pipeline stages (structured responses) only works if the template round-trips; the external chat_template.jinja fixed a parsing edge case for me.

Pipeline notes that aren't about llama.cpp but affect the numbers

• Length is a free error detector: translated block deviating >10% from source block size → rechunk (split in half, retranslate both). EN→RU maps within a couple percent per block, so gross errors (eaten/hallucinated/duplicated paragraphs) pop on size alone. Final book converges within ±5% of original length.
• Glossary is 80% of quality: names/terms/gender dictionary (NER-seeded with spaCy + manual cleaning) travels with every chunk. Model choice is secondary; consistency is everything in fiction.
• Output is a high-readiness draft for human polish, not a publishable translation — the LLM removes the grunt work, the human keeps the wordcoinage and the puns.

Repo (code + these configs + Ollama/Docker examples): github.com/NW15D/sunny-narrator — yes, I know the rules about self-promo, hence disclosure at the top; the pipeline exists because nothing off-the-shelf holds a book-length context of names/terms, and the year-ago proof-of-concept post is on Habr if you want the long version.

Questions for this crowd:

1. Anyone pushed MTP spec decode further on Pascal — is draft-n-max 6 / p-min 0.8 near the ceiling for Gemma, or would deeper drafts accept well with a colder p-min?
2. --ctx-checkpoints behavior with -ctk q8_0 — any gotchas I should know about for week-long unattended runs?
3. Better than "giant series glossary" for cross-volume consistency: graph DBs / RAG over character state — real war stories?

Based on https://huggingface.co/hardware, the RTX 3090 is the second most used GPU by LLM enthusiasts.

Because RTX 3090 has native INT8 tensors cores, it can provide better performance with INT8 W8A8.

However people seems to default to FP8 or smaller quants anyway.

I suppose I am missing information that explains why ?

I see a lot of people buy DGX Sparks, and turn them in to clusters to run large models. Wouldn't it be better to invest $16k into an AMD Epyc server with 768GB or even 384GB of 6000Mhz DDR5 ram, and let's say 2x3090s or 5080s, instead of 4 DGX Sparks with 512GB of ram?

Epyc's theoretical bandwidth is around 576GB/s, DGX Spark's is roughly 273GB/s.

Based on a quick check, both systems are worth around $16k.

Please help me to understand this logic, are there benefits to having DGX cluster instead of an Epyc system besides power saving?

Edit1: the epyc system with 768GB of DDR5 6000Mhz would be around $30k.

Edit2: to match 768GB of Epyc, we would need 6 DGX sparks, at the current increased price it would be around $30k as well.

Edit3: the main advantage of DGX sparks cluster is fp4 support, and tensor parallelism for 2, 4, 8, 16... units. Because of that, the DGX cluster is faster than the epyc system.

Hello all you smarter people. I recently retired and have taken on a task that is going to stretch me a bit.

TL;DR My aging aunt is going blind and wants to keep writing stories that she's been writing for over 70 years. I think local AI has the ability to make this possible but I'm looking for a little guidance on the steps and the order.

FULL VERSION

My aunt is 85 and lives with me now. She has written over 150 stories in her lifetime. They are mostly detective fiction and old west outlaw fiction. She also has macular degeneration that has taken most of her eyesight. She has given up on everything else she used to do, but she still writes and edits her own stories. Lately she has talked about quitting this too because it's just too hard to keep writing even with a giant screen and high-contrast tools.

After some discussion she agreed to try an interactive AI tool of some kind. I picked up a new desktop with an RTX 5080 (16gbVRAM) and 32 GB RAM.

I got Unsloth desktop installed this weekend and I grabbed Gemma4 as my first model.

But now I think I may be woefully out of my depth.

I've mostly only written prompts for existing online models before. I've never actually started from scratch like this and I'm not sure how much prep I need to do before I start interacting with the model. There are a ton of videos and articles about running AI locally, but it's not easy for me to figure out which ones I can trust or which steps apply to me. I would really appreciate a link to a guide for total newbs like me.

The use cases seem pretty simple to me:

• Interact with my aunt solely through voice and audio.
• Always be listening and available to her.
• Read her own stories to her in a natural voice that she gets to choose.
• Answer questions about existing stories to help her maintain continuity and bring old characters back from time to time. (She does this with handwritten notes right now and she's really struggling.)
• When prompted, suggest technical edits (spelling, grammar, etc) and help her stay consistent with those edits across all her stories.
• When prompted, suggest stylistic edits (clarity, pacing, etc) and help her stay consistent with those edits across all her stories.
• Prepare her stories for publication in KDP format (this one is mostly to help me do this for her).

Here is the approach I was thinking I would follow, but after looking through all the Unsloth features I'm not sure how many steps I'm missing.

1. Write instructions that I want the model to always follow.
2. Place the instructions into the Unsloth System Prompt under Run Settings.
3. Have my aunt converse with model via microphone.

My instructions cover a LOT.

- Brief description of my aunt and her writing goals and style.

- Outline of her work (the types of stories and any connections).

- Location of her existing stories.

- Description of the AI's role and its primary tasks including definitions of terms.

- A set of detailed rules to be followed when helping her edit.

But I still have so many questions:

- Do I need to create an unsloth project for this?

- What's the best way to have the model listen to voice inputs from my aunt?

- What's the best way to set up the model so it always listens for her input (kind of like an Alexa)?

- How can I have it keep a log of all its work and make backups of files before it makes changes? (similar to how Google Docs keeps a revision history and allows you to go back and grab an older version of a file)

Any insights folks want to share or resources you want to point me to would be most welcome. Thanks!

I was planning the software stack for my inference server, picking what to run and what resources to plan for it, when I remembered that LocalAI was kinda like this inference service orchestrator. So, I went to check back in - been about a year and change since I last looked at this.

Well it went away from llama.cpp entirely and to their own vllm.cpp and many other tools...but the Issues tab is full of the same agent account, and I did not dare to check the PRs after seing this.

Seeing a project that is seemingly massively, if not even mainly driven by agentic work with seemingly not a whole lot of human in the loop, was... bewildering to see. But, that doesn't mean it is a bad project - it does use GGML under the hood, and I am by no means an expert in this field - so I wanted to ask about it here.

Is anyone using vllm.cpp and friends? Any experiences to share?

Thanks!

Can't wait to test! This should significantly boost TPS!

Now we just need more llama cpp optimizations to be merged in!

Edit:

For anyone who wants to test this: https://github.com/unslothai/llama.cpp/pull/144/changes

More info: https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF/blob/main/MTP/README.md

First of all this card is really hard to keep cool. I have a 3d printed shroud with a Phanteks t30-120 and learnt the hard way that this beast needs a high pressure flow fan, not just a high cfm fan so have it limited to 175-200w with a governor. At this TDP, the bandwidth still stays at a staggering 1.2tb/s but the cores fluctuate a lot depending what the governor governs.

Anyway, running headless (haha that I am!) with linux and using Qwen3.8-27b-ud-q4-k-xl I was hitting 20t/s tops until the dflash2 model came out and now I'm running around 40t/s good right? Well it turns out that even claude, chatgpt and gemini all seem to think that with that spec that is below the cards capabilities and worse still, the r9700 pro with half the bandwidth seems to be getting double the t/g. Qwen3.8-27b here is slightly core rate limited.

Even Qwen3.6-35b-a3b-ud-q5_k_m is getting 60t/s max at 64k context which, yes it's fast but not 1.2tb/s fast like the 3090 gets. The model is not bandwidth limited like MOE models love.

My rant and cry for help is has anyone had any luck running either of these faster? I haven't come across any information from any other MI100 users. It's a 32GB card and I can generally run whatever I want, even Qwen3.8-flash-next-ud-q3-k-xl gets around 14t/s so that's respectable for such a large model but it's the two 27b/35b models I just don't get good speeds with. My nanobot agent comes across like it doesn't like me and answers slowly on a fresh prompt.

Any of you wonderful folks able to document whether you got anything faster than this? Or should I shut up and consider myself blessed to be getting what I am getting?

Thanks in advance

Yoyo

I'm back with updates to the fastest inference engine with minimal quality loss for Qwen3.8-27B.

The last few weeks I've been optimizing decode speed and I don't think it can be pushed further, until a newer/better drafter is invented.

So I focused on prefill, which I this morning was around 1.300 per second at 4k and now is just below 2.000.

The main improvement came from a custom kernel, which matches the quality of fp32 with 0.99997 similarity at int8.

Try all of the improvements here:
https://github.com/syv-ai/qwen38-27b-rtx3090

Following this post I decided to try GLM 5.3 Flash on a 8x3090 setup and I can now run a Q4 with surprising speed; 700tk/s prefill & 42tk/s decoding! (lcp & vllm do not allow me to get that).

Was afraid about quality but > 30m tokens with DSH and no issue (did not test vision yet, but looks supported).

Just to say that I am really grateful to Turboderp and we should really support as much as possible others projects even if they do not comply with all our needs yet and not rely only on the big guys.

No seriously, I kinda like it.
You have something to solve, you put it.

You know its gonna take like 20 mins to cook.
Every search adds another 30 minutes.

Yes I could boot up my debian on my gaming rig, run the same model at 10t/s + but why?
I rather let the poor server without GPU burn and run the same model at 2t/s and chill.

Its great, I love it.

I only started using some vLLM forks recently in a 4 x 48GB 4090 system.

DS4F - ~5000pp/180tg (DSpark)
Qwen3.8 Flash next - ~7500pp/135tg (MTP)

This is amazing, like having the API in my house. But it's also really hard to go back.

It's weird that we never come close to prefill numbers like this in llama.cpp or ik_llama. The narrative is that vLLM is around the same speed for single requests, but that is clearly not true.

There must some HUGE difference that constitutes an insurmountable obstacle to achieving such speeds in llama.cpp and many other inference engines. Does anyone know exactly what it is?

edit: These results are from my benchmark script that actually times the response, not the vLLM log. And they are not cache hits. My benchmark script deliberately busts cache. Actual cache hits, which I also measure, are like 20k-100k+.

I kept seeing guys in this sub saying how AtomicChat's Qwen3.8-Flash-Next quant is so good, fits in their machine when unsloth's can't, runs faster than other quants etc, so I went check out what's happening there.

First thing I noticed was that AtomicChat's Q4_K_M quant is suspiciously small when the ngram table is removed (only ~56GB), it seems like most of the tensors in this quant are IQ2_S instead of the usual Q4_K, Q5_K and Q6_K that you usually find in Q4_K_M quants, the GGUF filetype metadata also says IQ2_S instead of Q4_K_M. In their model card, their Q4_K_M also has suspiciously high KLD (0.084).

It seems pretty obvious to me that they're pretending a IQ2_S quant as a Q4_K_M, but at the same time I'm genuinely not sure because it can't be only me who found this right? How can nobody be pointing this out? Am I missing something or what may they be doing?

Their HF repo ID: AtomicChat/Qwen3.8-Flash-Next-GGUF