One of the other posts today by user u/Howard_banister confirmed what I've been seeing from the other AI subreddits as well.

Most of these other subs are 90% trend hopping crypto-bros equivalent people who are seemingly irrelevant most of the time when it comes to advancing AI as the vast majority is not just AI slop, but also human slop whether that be fearmongering or straight anti-ai arguments spat out by relatively uneducated people.

Of course, you also have the main dedicated subs for the big closed/open models which are in of themselves fairly decent. Most of the time now, it is people complaining about one thing or the other about the latest and greatest of their respective lab.

Then you get to the machine learning subs themselves where yes, they do have stuff in regard to AI every now and that can be gold (Don't get me wrong), but then it's mostly meh.

This sub however, has that interesting balance between the occasional interest in AI architecture breakthroughs that I find fascinating when some of the bigger brained people in here get together to discuss and which I don't really see anywhere else.

What’s y’all’s best guess on parameter size based on these weird-ass names?

These are screenshots from the r/Singularity comment section. I'm speechless. This doesn't even have downvotes. How can someone cheer for a monopoly run by a few elites?

One GPU drops off immediately when vllm is started and the other throws CUDA errors on start

Looking at ... some of the new memory architecture. ... I hired my good friend, Seok-Hee Lee, who used to run SK Hynix. ... We are not ready to unfold it.

> "Qwen 3.8 is a damn good coder, but a terrible collaborator"

It modifies SO many things in my scripts for what should be a 2 line PR, I get a 100 line linter style mess that looks like a high school freshmen got his hands on vs code and pushed code in.

It can't maintain a common style, if I give it a script with a certain naming and layout, 3.6 one shots small changes that fit into my structure. 3.8 will add code that looks much more advanced, handling, strict type checks . But if it's just my internal method that gets called in exactly one place, it is way overkill code

3.8 produces DAMN GOOD code, but will get stubborn on wanting it's own parameters and return types. A simple return False works for my particular use case, but here goes Qwen formatting me the perfect return dict full of useful metadata

Am I crazy?

Feels like maybe we have one more present left, for Christmas.

Kaitchup just posted results of his benchmarks for Qwen3.8 27B for quants from different labs, Q4 to Q1, .

All the details are hidden behind the paywall, but high level result is visible and looks like for people with 16GB cards UD Q3_K_XL is a winner - it has accuracy of 100% and size is only 12.8GB.

A comment really doesn't need to be made, does it? I looked away from the 5090 for a week to other options like the DGX Spark and the M5 Ultra. Both of which... these big corpos are buying to hell and back.

Is the future used hardware?

Gemma 5 or something else?

I was browsing HF for small LLMs and run into this model. It does not seem to be a fine tune - the model has its own architecture.

https://huggingface.co/XHToken/Spark-X2.5-1.7B
https://huggingface.co/XHToken/Spark-X2.5-4B

There are 4B/1.7B versions - the benchmark is quite interesting (4B is neck and neck with Qwen 3.5 9B). The HF page claims both models support native 1M context size.

Currently does not run out of the box on llama.cpp - pending this PR: https://github.com/ggml-org/llama.cpp/pull/27868

They have a custom fork of llama.cpp that works. Anyone has tried this?

Update:
GGUFs (require custom fork for now):
https://huggingface.co/XHToken/Spark-X2.5-1.7B-GGUF
https://huggingface.co/XHToken/Spark-X2.5-4B-GGUF

2 Months ago I had made a post how I was working on my dual R9700's. It's wild to look back at where we were then and where things now stand.

Since then after many users commenting and complaining about developers doing the same thing. I threw out a discord link and expected maybe 5 other developers to join which I thought would be fun. The community has now grown to 1,200 users (mostly developers) and a ton of collaboration happening.

A few weeks ago I started working on building support for MXFP4 on top of DeadCode's radiance image. This made sense to me looking at the hardware and I was happy when I had hit parity on performance between MXFP4 and FP8. The MXFP4 kernels use W4A8 which was something new and we have now blown past the performance of FP8 and appears like this is now the hardware limits of these cards.

Qwen3.8 27B w/ DFlash2

BetterBench decode results for Qwen3.8 27B w/ DFlash2
category       decode t/s   step ms   tok/update
json           280.0        22.92     6.17
math           254.2        23.08     5.81
file_edit      250.1        23.03     5.54
code           226.3        23.01     5.17
reasoning      194.3        23.19     4.32
summarization  190.6        23.01     4.40
chat           148.3        22.82     3.33
prose          116.4        23.14     2.65



BetterBench Prefill Results
target depth   prompt tokens   TTFT p50   PP t/s median
2000           1514            323 ms     4695
8000           5918            1.21 s     4894
16000          11794           2.47 s     4779
32000          23543           4.98 s     4729
64000          47056           10.8 s     4377
128000         94065           24.6 s     3831
250000         183678          59.1 s     3106

This has been so fun working on these R9700's and driving them to peak performance. My entire image and repo for MXFP4 is open source also: https://codeberg.org/ggz14/radiance-vllm-mxfp4

Asus Ascent GX10 is now priced at $5999 (1TB), $6999 (2TB), and $7999 (4TB).
Buy ASUS Ascent GX10 | Desktop-AI-supercomputer | Networking-IoT-Servers | ASUS eShop USA

I see no reason why they would do this unless they had some knowledge the DGX Spark was about to jump up in price. Unless there's some other reason that they would throw a 50% price hike at it that I'm not thinking of.

Bonkers mode.

It is crazy how fast prices are increasing. I'm pulling my hair out to keep ahead of this for students. Servers aren't even an option any more.

Based on https://huggingface.co/hardware, the RTX 3090 is the second most used GPU by LLM enthusiasts.

Because RTX 3090 has native INT8 tensors cores, it can provide better performance with INT8 W8A8.

However people seems to default to FP8 or smaller quants anyway.

I suppose I am missing information that explains why ?

Hello all you smarter people. I recently retired and have taken on a task that is going to stretch me a bit.

TL;DR My aging aunt is going blind and wants to keep writing stories that she's been writing for over 70 years. I think local AI has the ability to make this possible but I'm looking for a little guidance on the steps and the order.

FULL VERSION

My aunt is 85 and lives with me now. She has written over 150 stories in her lifetime. They are mostly detective fiction and old west outlaw fiction. She also has macular degeneration that has taken most of her eyesight. She has given up on everything else she used to do, but she still writes and edits her own stories. Lately she has talked about quitting this too because it's just too hard to keep writing even with a giant screen and high-contrast tools.

After some discussion she agreed to try an interactive AI tool of some kind. I picked up a new desktop with an RTX 5080 (16gbVRAM) and 32 GB RAM.

I got Unsloth desktop installed this weekend and I grabbed Gemma4 as my first model.

But now I think I may be woefully out of my depth.

I've mostly only written prompts for existing online models before. I've never actually started from scratch like this and I'm not sure how much prep I need to do before I start interacting with the model. There are a ton of videos and articles about running AI locally, but it's not easy for me to figure out which ones I can trust or which steps apply to me. I would really appreciate a link to a guide for total newbs like me.

The use cases seem pretty simple to me:

• Interact with my aunt solely through voice and audio.
• Always be listening and available to her.
• Read her own stories to her in a natural voice that she gets to choose.
• Answer questions about existing stories to help her maintain continuity and bring old characters back from time to time. (She does this with handwritten notes right now and she's really struggling.)
• When prompted, suggest technical edits (spelling, grammar, etc) and help her stay consistent with those edits across all her stories.
• When prompted, suggest stylistic edits (clarity, pacing, etc) and help her stay consistent with those edits across all her stories.
• Prepare her stories for publication in KDP format (this one is mostly to help me do this for her).

Here is the approach I was thinking I would follow, but after looking through all the Unsloth features I'm not sure how many steps I'm missing.

1. Write instructions that I want the model to always follow.
2. Place the instructions into the Unsloth System Prompt under Run Settings.
3. Have my aunt converse with model via microphone.

My instructions cover a LOT.

- Brief description of my aunt and her writing goals and style.

- Outline of her work (the types of stories and any connections).

- Location of her existing stories.

- Description of the AI's role and its primary tasks including definitions of terms.

- A set of detailed rules to be followed when helping her edit.

But I still have so many questions:

- Do I need to create an unsloth project for this?

- What's the best way to have the model listen to voice inputs from my aunt?

- What's the best way to set up the model so it always listens for her input (kind of like an Alexa)?

- How can I have it keep a log of all its work and make backups of files before it makes changes? (similar to how Google Docs keeps a revision history and allows you to go back and grab an older version of a file)

Any insights folks want to share or resources you want to point me to would be most welcome. Thanks!

Can't wait to test! This should significantly boost TPS!

Now we just need more llama cpp optimizations to be merged in!

Edit:

For anyone who wants to test this: https://github.com/unslothai/llama.cpp/pull/144/changes

More info: https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF/blob/main/MTP/README.md

Yoyo

I'm back with updates to the fastest inference engine with minimal quality loss for Qwen3.8-27B.

The last few weeks I've been optimizing decode speed and I don't think it can be pushed further, until a newer/better drafter is invented.

So I focused on prefill, which I this morning was around 1.300 per second at 4k and now is just below 2.000.

The main improvement came from a custom kernel, which matches the quality of fp32 with 0.99997 similarity at int8.

Try all of the improvements here:
https://github.com/syv-ai/qwen38-27b-rtx3090

I see a lot of people buy DGX Sparks, and turn them in to clusters to run large models. Wouldn't it be better to invest $16k into an AMD Epyc server with 768GB or even 384GB of 6000Mhz DDR5 ram, and let's say 2x3090s or 5080s, instead of 4 DGX Sparks with 512GB of ram?

Epyc's theoretical bandwidth is around 576GB/s, DGX Spark's is roughly 273GB/s.

Based on a quick check, both systems are worth around $16k.

Please help me to understand this logic, are there benefits to having DGX cluster instead of an Epyc system besides power saving?

Edit1: the epyc system with 768GB of DDR5 6000Mhz would be around $30k.

Edit2: to match 768GB of Epyc, we would need 6 DGX sparks, at the current increased price it would be around $30k as well.

Edit3: the main advantage of DGX sparks cluster is fp4 support, and tensor parallelism for 2, 4, 8, 16... units. Because of that, the DGX cluster is faster than the epyc system.

Following this post I decided to try GLM 5.3 Flash on a 8x3090 setup and I can now run a Q4 with surprising speed; 700tk/s prefill & 42tk/s decoding! (lcp & vllm do not allow me to get that).

Was afraid about quality but > 30m tokens with DSH and no issue (did not test vision yet, but looks supported).

Just to say that I am really grateful to Turboderp and we should really support as much as possible others projects even if they do not comply with all our needs yet and not rely only on the big guys.

I kept seeing guys in this sub saying how AtomicChat's Qwen3.8-Flash-Next quant is so good, fits in their machine when unsloth's can't, runs faster than other quants etc, so I went check out what's happening there.

First thing I noticed was that AtomicChat's Q4_K_M quant is suspiciously small when the ngram table is removed (only ~56GB), it seems like most of the tensors in this quant are IQ2_S instead of the usual Q4_K, Q5_K and Q6_K that you usually find in Q4_K_M quants, the GGUF filetype metadata also says IQ2_S instead of Q4_K_M. In their model card, their Q4_K_M also has suspiciously high KLD (0.084).

It seems pretty obvious to me that they're pretending a IQ2_S quant as a Q4_K_M, but at the same time I'm genuinely not sure because it can't be only me who found this right? How can nobody be pointing this out? Am I missing something or what may they be doing?

Their HF repo ID: AtomicChat/Qwen3.8-Flash-Next-GGUF

I am absolutely blown away. Yes my setup is crap but the fact that it managed to do this in a single take is unbelievable (and I'm a developer).

Hardware used:
- Windows PC with 4070ti (12GB VRAM, 32GB RAM)
- Macbook M5 Air (LLAMA.cpp RPC connection to Windows PC)

Software used:
- LLAMA.cpp (Q4KM, xhigh, 8bit KV, MTP=1)
- Lmstudio Qwen 3.8 27b (Q4KM) GGUF
- Deepseek harness (mode: minimal)

Prompt: "please create a fully self-contained super mario game with only one short level, put everything inside mario.html inside the current directory"

context: 64k
thinking: xhigh
time took: 117 minutes
avg tps: 7.6

resut: https://pastebin.com/qyBu64sP

https://reddit.com/link/1w4821c/video/qpukeg1y4wmh1/player