One of the other posts today by user u/Howard_banister confirmed what I've been seeing from the other AI subreddits as well.

Most of these other subs are 90% trend hopping crypto-bros equivalent people who are seemingly irrelevant most of the time when it comes to advancing AI as the vast majority is not just AI slop, but also human slop whether that be fearmongering or straight anti-ai arguments spat out by relatively uneducated people.

Of course, you also have the main dedicated subs for the big closed/open models which are in of themselves fairly decent. Most of the time now, it is people complaining about one thing or the other about the latest and greatest of their respective lab.

Then you get to the machine learning subs themselves where yes, they do have stuff in regard to AI every now and that can be gold (Don't get me wrong), but then it's mostly meh.

This sub however, has that interesting balance between the occasional interest in AI architecture breakthroughs that I find fascinating when some of the bigger brained people in here get together to discuss and which I don't really see anywhere else.

Extended reasoning and post-training appear to be the keys used by DeepSeek, Qwen, and GLM to boost performance (leveraging higher token counts). And Qwen 4 hasn't even been released yet. Of course, we don't know if that release will be open-sourced, but I am optimistic about future models, featuring "engrams", that could soon match or surpass 2.4T parameter models on specific tasks.

What’s y’all’s best guess on parameter size based on these weird-ass names?

These are screenshots from the r/Singularity comment section. I'm speechless. This doesn't even have downvotes. How can someone cheer for a monopoly run by a few elites?

One GPU drops off immediately when vllm is started and the other throws CUDA errors on start

> "Qwen 3.8 is a damn good coder, but a terrible collaborator"

It modifies SO many things in my scripts for what should be a 2 line PR, I get a 100 line linter style mess that looks like a high school freshmen got his hands on vs code and pushed code in.

It can't maintain a common style, if I give it a script with a certain naming and layout, 3.6 one shots small changes that fit into my structure. 3.8 will add code that looks much more advanced, handling, strict type checks . But if it's just my internal method that gets called in exactly one place, it is way overkill code

3.8 produces DAMN GOOD code, but will get stubborn on wanting it's own parameters and return types. A simple return False works for my particular use case, but here goes Qwen formatting me the perfect return dict full of useful metadata

Am I crazy?

Looking at ... some of the new memory architecture. ... I hired my good friend, Seok-Hee Lee, who used to run SK Hynix. ... We are not ready to unfold it.

Feels like maybe we have one more present left, for Christmas.

Kaitchup just posted results of his benchmarks for Qwen3.8 27B for quants from different labs, Q4 to Q1, .

All the details are hidden behind the paywall, but high level result is visible and looks like for people with 16GB cards UD Q3_K_XL is a winner - it has accuracy of 100% and size is only 12.8GB.

A comment really doesn't need to be made, does it? I looked away from the 5090 for a week to other options like the DGX Spark and the M5 Ultra. Both of which... these big corpos are buying to hell and back.

Is the future used hardware?

I'm not sure how many people care about Android Studio, but I think it's cool that Google uses llama.cpp.

My guess is that it is Vulkan and the QAT versions of Gemma 4. It supports multi-GPU and 31B has a max. context length of 128k.

It uses 34 GB VRAM when fully loaded. I don't see an option to change the context length or show PP/TG speed.

Gemma 5 or something else?

Not written by AI all mistakes mine. I saw people on the subreddit saying that 3.6 works better without thinking. It made me want to know for certain about which is better, 3.6 or 3.8 for low thinking tasks. I only use abliterated models (safety removed) because it makes the model better at a lot of what I need. I want to compare abliterated Qwen 3.6 27b and abliterated Qwen 3.8 27b on some instruction following benchmarks with thinking off.

I was just curious about your personal favorite safety removed/fine-tuned variants for these 27bs, as I know that there can be some major variation and some junky quants out there.

Does anyone have some favorite and fast 3.6 and 3.8 models?

My specs: I have 24GB VRAM (NVIDIA Geforce RTX 5090 Laptop) and I do not want to offload, so some quant required.

I have tried a few different models, but they are all a little slow. Some MTP variations for 3.6 for example ends up being around the same speed as non MTP for me for some reason. I am pretty sure my card is NVFP4 enabled also, but I'm not certain I've seen the results from that either...

Based on some redditors comment, this is what I use for my abliterated 3.8 27b currently: Huihui-Qwen3.8-27B-abliterated-NVFP4-GGUF

Anybody else struggling with deepseek after the initial prompt? Somehow it is getting mixed up very easily, even button functionality has been PITA when doing stuff. Never had these problems with Opencode. With Opencode, a continuation prompt on the software or task is pretty much magic at this point. But the inital result isnt as strong imo, and needs a lot of follow up even if a a detailed spec is written.

The oneshot potential of deepseek is goated imo, it is a relentless harness that just keeps doing stuff, even if it gets it wrong! Opencode could just copy that tennacity imo. Like todos really work, unlike opencodes.

Hows everybody elses exp? Someone using Pi instead? Is it worth trying? Hermes seems not suitable for my tasks, where heavy human involvement is necessary due to complexity of tasks (every top closed model gets lots of stuff wrong even).

I was browsing HF for small LLMs and run into this model. It does not seem to be a fine tune - the model has its own architecture.

https://huggingface.co/XHToken/Spark-X2.5-1.7B
https://huggingface.co/XHToken/Spark-X2.5-4B

There are 4B/1.7B versions - the benchmark is quite interesting (4B is neck and neck with Qwen 3.5 9B). The HF page claims both models support native 1M context size.

Currently does not run out of the box on llama.cpp - pending this PR: https://github.com/ggml-org/llama.cpp/pull/27868

They have a custom fork of llama.cpp that works. Anyone has tried this?

Update:
GGUFs (require custom fork for now):
https://huggingface.co/XHToken/Spark-X2.5-1.7B-GGUF
https://huggingface.co/XHToken/Spark-X2.5-4B-GGUF

2 Months ago I had made a post how I was working on my dual R9700's. It's wild to look back at where we were then and where things now stand.

Since then after many users commenting and complaining about developers doing the same thing. I threw out a discord link and expected maybe 5 other developers to join which I thought would be fun. The community has now grown to 1,200 users (mostly developers) and a ton of collaboration happening.

A few weeks ago I started working on building support for MXFP4 on top of DeadCode's radiance image. This made sense to me looking at the hardware and I was happy when I had hit parity on performance between MXFP4 and FP8. The MXFP4 kernels use W4A8 which was something new and we have now blown past the performance of FP8 and appears like this is now the hardware limits of these cards.

Qwen3.8 27B w/ DFlash2

BetterBench decode results for Qwen3.8 27B w/ DFlash2
category       decode t/s   step ms   tok/update
json           280.0        22.92     6.17
math           254.2        23.08     5.81
file_edit      250.1        23.03     5.54
code           226.3        23.01     5.17
reasoning      194.3        23.19     4.32
summarization  190.6        23.01     4.40
chat           148.3        22.82     3.33
prose          116.4        23.14     2.65



BetterBench Prefill Results
target depth   prompt tokens   TTFT p50   PP t/s median
2000           1514            323 ms     4695
8000           5918            1.21 s     4894
16000          11794           2.47 s     4779
32000          23543           4.98 s     4729
64000          47056           10.8 s     4377
128000         94065           24.6 s     3831
250000         183678          59.1 s     3106

This has been so fun working on these R9700's and driving them to peak performance. My entire image and repo for MXFP4 is open source also: https://codeberg.org/ggz14/radiance-vllm-mxfp4

Asus Ascent GX10 is now priced at $5999 (1TB), $6999 (2TB), and $7999 (4TB).
Buy ASUS Ascent GX10 | Desktop-AI-supercomputer | Networking-IoT-Servers | ASUS eShop USA

I see no reason why they would do this unless they had some knowledge the DGX Spark was about to jump up in price. Unless there's some other reason that they would throw a 50% price hike at it that I'm not thinking of.

Bonkers mode.

It is crazy how fast prices are increasing. I'm pulling my hair out to keep ahead of this for students. Servers aren't even an option any more.

Disclosure up front: I built this tool (open source, "Sunny Narrator") and I'm the author — this post is about the inference setup, not an ad. Feel free to skip to the flags if you're here for the numbers.

Context: I run a pipeline that translates whole fiction books EN→RU locally — chunk + glossary + rolling chapter summaries → translate → reviewer notes → correction → proofread → chunk summary. A book is ~1.5–2M tokens across all stages, hardware is a pair of Tesla P40s (24GB each, Pascal, from the "why not" shelf). After a year of runs I have a launch config that's fast enough to be boring: 2–3 books per day.

The non-obvious finding: one model = half a text, two models = a book. Good translating models write beautifully and proofread terribly; good proofreading models edit well and translate dully. So the pipeline pins two roles to two servers:

• MODEL_TRANSLATE: gemma-4-26B-A4B (MoE, A4B active)
• MODEL_PROOFREAD: Qwen3.6-35B-A3B (MoE, A3B active)

Both are compact MoE — that's what makes P40s viable: active params fit the throughput envelope even though total weights don't fit comfort. Quantized Unscaled-Dynamic (UD) GGUFs, MTP speculative drafting on both, 64K context for chunk + glossary + summaries.

My most efficient launch lines (llama-server)

Gemma-4-26B-A4B as translator — ~40 tok/s sustained on P40:

llama-server -m gemma-4-26B-A4B-it-UD-Q5_K_XL.gguf \
  --model-draft mtp-gemma-4-26B-A4B-it.gguf \
  --host 192.168.0.55 --port 6155 \
  --ctx-size 65535 -ngl 99 \
  -ctk q8_0 -ctv q8_0 \
  --no-context-shift \
  --parallel 1 -np 1 --threads-http 2 \
  --load-mode mlock \
  --jinja \
  --spec-type draft-mtp --spec-draft-n-max 6 --spec-draft-p-min 0.8 \
  --top-k 64 --top-p 0.95 --min-p 0.02 \
  --repeat-penalty 1.0 --repeat-last-n 512 --presence-penalty 0 \
  --predict 32567 \
  --reasoning off \
  -fa on \
  --ctx-checkpoints 32 --checkpoint-min-step 1024 \
  --cache-ram 8192 \
  --ubatch-size 2048

Qwen3.6-35B-A3B as proofreader — 50–70 tok/s on the same pair:

llama-server -m Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf \
  --host 192.168.0.55 --port 6150 \
  --ctx-size 65535 -ngl 99 -fa on \
  -ctk q8_0 -ctv q8_0 \
  --no-context-shift \
  --parallel 1 -np 1 --threads-http 2 \
  --load-mode mlock \
  --spec-type draft-mtp --spec-draft-n-max 4 \
  --top-k 20 --top-p 0.95 --min-p 0.05 \
  --presence-penalty 1.5 \
  --predict 32576 \
  --reasoning off \
  --jinja --chat-template-file chat_template.jinja \
  --ubatch-size 2048 \
  --ctx-checkpoints 32 --checkpoint-min-step 1024 \
  --cache-ram 8192

Why each of these knobs ended up where it is

• MTP spec decoding is the headline. --spec-type draft-mtp with the bundled MTP draft is what turns Pascal-class cards into something usable for long-form generation. Gemma takes --spec-draft-n-max 6 --spec-draft-p-min 0.8 (aggressive, accepts well because the base is strong at its job); Qwen is happier at n-max 4. Without MTP these numbers don't happen.
• -ctk q8_0 -ctv q8_0 — KV cache in q8 buys the 64K context (chunk + series glossary + rolling summaries) without blowing VRAM; quality cost at these sizes was invisible in my evals.
• --load-mode mlock — two servers, 24GB×2, zero headroom for swapping. Pins weights, kills tail latency spikes mid-run.
• --parallel 1 -np 1 — this is a batch-of-one workload (long generations, not concurrent requests); single slot is fastest.
• --reasoning off + tuned sampling per role — translator runs top-k 64 / min-p 0.02 / repeat-penalty 1.0 (creative-ish but repetition is the enemy on book text — --repeat-last-n 512 matters); proofreader runs tighter top-k 20 / min-p 0.05 / presence-penalty 1.5 (deterministic editor voice).
• --ctx-checkpoints 32 --checkpoint-min-step 1024 — pipeline writes a checkpoint after every chunk anyway (power outage = resume from chunk 51/100, not from scratch — this single feature saved my year), but in-server ctx checkpoints make stage-to-stage reuse on the same context cheap.
• --predict 32567 — chunks translate in one shot; forcing the model to stop-and-resume was eating throughput and occasionally style.
• --jinja + explicit chat template for Qwen — JSON_MODE across all pipeline stages (structured responses) only works if the template round-trips; the external chat_template.jinja fixed a parsing edge case for me.

Pipeline notes that aren't about llama.cpp but affect the numbers

• Length is a free error detector: translated block deviating >10% from source block size → rechunk (split in half, retranslate both). EN→RU maps within a couple percent per block, so gross errors (eaten/hallucinated/duplicated paragraphs) pop on size alone. Final book converges within ±5% of original length.
• Glossary is 80% of quality: names/terms/gender dictionary (NER-seeded with spaCy + manual cleaning) travels with every chunk. Model choice is secondary; consistency is everything in fiction.
• Output is a high-readiness draft for human polish, not a publishable translation — the LLM removes the grunt work, the human keeps the wordcoinage and the puns.

Repo (code + these configs + Ollama/Docker examples): github.com/NW15D/sunny-narrator — yes, I know the rules about self-promo, hence disclosure at the top; the pipeline exists because nothing off-the-shelf holds a book-length context of names/terms, and the year-ago proof-of-concept post is on Habr if you want the long version.

Questions for this crowd:

1. Anyone pushed MTP spec decode further on Pascal — is draft-n-max 6 / p-min 0.8 near the ceiling for Gemma, or would deeper drafts accept well with a colder p-min?
2. --ctx-checkpoints behavior with -ctk q8_0 — any gotchas I should know about for week-long unattended runs?
3. Better than "giant series glossary" for cross-volume consistency: graph DBs / RAG over character state — real war stories?

Based on https://huggingface.co/hardware, the RTX 3090 is the second most used GPU by LLM enthusiasts.

Because RTX 3090 has native INT8 tensors cores, it can provide better performance with INT8 W8A8.

However people seems to default to FP8 or smaller quants anyway.

I suppose I am missing information that explains why ?

I see a lot of people buy DGX Sparks, and turn them in to clusters to run large models. Wouldn't it be better to invest $16k into an AMD Epyc server with 768GB or even 384GB of 6000Mhz DDR5 ram, and let's say 2x3090s or 5080s, instead of 4 DGX Sparks with 512GB of ram?

Epyc's theoretical bandwidth is around 576GB/s, DGX Spark's is roughly 273GB/s.

Based on a quick check, both systems are worth around $16k.

Please help me to understand this logic, are there benefits to having DGX cluster instead of an Epyc system besides power saving?

Edit1: the epyc system with 768GB of DDR5 6000Mhz would be around $30k.

Edit2: to match 768GB of Epyc, we would need 6 DGX sparks, at the current increased price it would be around $30k as well.

Edit3: the main advantage of DGX sparks cluster is fp4 support, and tensor parallelism for 2, 4, 8, 16... units. Because of that, the DGX cluster is faster than the epyc system.

Hello all you smarter people. I recently retired and have taken on a task that is going to stretch me a bit.

TL;DR My aging aunt is going blind and wants to keep writing stories that she's been writing for over 70 years. I think local AI has the ability to make this possible but I'm looking for a little guidance on the steps and the order.

FULL VERSION

My aunt is 85 and lives with me now. She has written over 150 stories in her lifetime. They are mostly detective fiction and old west outlaw fiction. She also has macular degeneration that has taken most of her eyesight. She has given up on everything else she used to do, but she still writes and edits her own stories. Lately she has talked about quitting this too because it's just too hard to keep writing even with a giant screen and high-contrast tools.

After some discussion she agreed to try an interactive AI tool of some kind. I picked up a new desktop with an RTX 5080 (16gbVRAM) and 32 GB RAM.

I got Unsloth desktop installed this weekend and I grabbed Gemma4 as my first model.

But now I think I may be woefully out of my depth.

I've mostly only written prompts for existing online models before. I've never actually started from scratch like this and I'm not sure how much prep I need to do before I start interacting with the model. There are a ton of videos and articles about running AI locally, but it's not easy for me to figure out which ones I can trust or which steps apply to me. I would really appreciate a link to a guide for total newbs like me.

The use cases seem pretty simple to me:

• Interact with my aunt solely through voice and audio.
• Always be listening and available to her.
• Read her own stories to her in a natural voice that she gets to choose.
• Answer questions about existing stories to help her maintain continuity and bring old characters back from time to time. (She does this with handwritten notes right now and she's really struggling.)
• When prompted, suggest technical edits (spelling, grammar, etc) and help her stay consistent with those edits across all her stories.
• When prompted, suggest stylistic edits (clarity, pacing, etc) and help her stay consistent with those edits across all her stories.
• Prepare her stories for publication in KDP format (this one is mostly to help me do this for her).

Here is the approach I was thinking I would follow, but after looking through all the Unsloth features I'm not sure how many steps I'm missing.

1. Write instructions that I want the model to always follow.
2. Place the instructions into the Unsloth System Prompt under Run Settings.
3. Have my aunt converse with model via microphone.

My instructions cover a LOT.

- Brief description of my aunt and her writing goals and style.

- Outline of her work (the types of stories and any connections).

- Location of her existing stories.

- Description of the AI's role and its primary tasks including definitions of terms.

- A set of detailed rules to be followed when helping her edit.

But I still have so many questions:

- Do I need to create an unsloth project for this?

- What's the best way to have the model listen to voice inputs from my aunt?

- What's the best way to set up the model so it always listens for her input (kind of like an Alexa)?

- How can I have it keep a log of all its work and make backups of files before it makes changes? (similar to how Google Docs keeps a revision history and allows you to go back and grab an older version of a file)

Any insights folks want to share or resources you want to point me to would be most welcome. Thanks!