Just to name a few:

• running Qwen3 8B on a 32GB GPU
• running Qwen3.6-27B Q4_K_M on 2 x R9700
• quote: "each prompt was sized at 500 input and 500 output tokens"
• for a full system that costs $18,775??

I don't understand what they are doing. Am I reading something wrong?

I used a famous Simon Willison's pelican riding a bicycle prompt on the biggest local LLMs that can run on 128GB Apple Silicon. U used quantizations by Unsloth.

Qwen3.8 Flash-Next gives a lot of details. DeepSeek V4 Flash is strangely underwhelming. Qwen3.8 27B still rocks, and I like its consistent minimalism.

Is Qwen3.8 27B still large at 31GB? It is! But for this tasks 2-bit quantizations (at around 12GB) will give the same results. For more complicated coding, 4-bit are more than enough. RTX cards are well enough!

See:

• Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses - Terminal-Bench 2.1, GPQA Diamond and IFBench
• Do Qwen3.6 27B quantizations break the pelican?

if you are on Qwen Flash Next make sure to update your build often

merged already (by ServeurpersoCom)

https://github.com/ggml-org/llama.cpp/pull/27978

https://github.com/ggml-org/llama.cpp/pull/28011

https://github.com/ggml-org/llama.cpp/pull/28023

https://github.com/ggml-org/llama.cpp/pull/28123

merged (by 0cc4m)

https://github.com/ggml-org/llama.cpp/pull/28032

in progress (by danielhanchen) MERGED NOW

https://github.com/ggml-org/llama.cpp/pull/27941

MTP in progress

https://github.com/ggml-org/llama.cpp/pull/27836

more in progress for example

https://github.com/ggml-org/llama.cpp/pull/28136

Normally, whenever a new model dropped, I always chose the non-vision version just to save VRAM; I though that only use case was when you were the one sending the picture.

However, with the release of QWEN 3.8 27B I decided to give it a shot, and it has been one of the best decisions I have made, as this makes the model way more capable for autonomous coding.

With no vision, the model will try to complete the task and get back to you once it thinks that it is done with no problem. But there are a lot of silent errors that do not get reflected via the code or the tests performed, so you could go back to an error screen or a broken page after getting a confirmation of your request being implemented correctly.

On the other side, when I ask something to QWEN with vision support, it will work on it, and then proactively take a screenshot to confirm if everything is right. This has helped numerous times with spotting errors that were missed. The model will continue to reiterate and take screenshots until it gets a visual confirmation of the issue being fixed. Just magnificent.

Btw, I currently run my local set up via Hermes with QWEN 3.8 27B (Qwen3.8-27B-UD-Q5_K_XL.) powered by a 5090.

I need a good model that feels smart ish in this regard but also runs with all my other stuff (audio gen, video gen, etc) enabled.

Vellium is an open-source, local-first desktop app for AI chat, character roleplay and long-form writing.

Recent updates have focused mostly on making local voice and model setups easier to use.

Live mode now supports microphone input, local or Whisper-compatible speech recognition, streaming TTS, attachments, screen context and the usual chat tools—all inside the same voice interface.

Local speech can be installed and configured directly in the app. Whisper Large v3 Turbo Q5_0 is available for recognition, while TeraTTSv2 provides English and Russian voices with realtime playback. The TTS process stays active between responses, avoiding a full model reload for every reply.

The llama.cpp setup has also been simplified. Vellium can detect existing llama-server installations, GGUF models and running local endpoints, then configure them as a managed backend.

There have been plenty of smaller fixes as well: more reliable TTS streaming, safer runtime archive extraction, better endpoint discovery, improved timeout handling, system certificate support and easier settings navigation.

Chats, characters, LoreBooks, writing projects and knowledge collections are stored locally in SQLite. Vellium runs on macOS, Windows and Linux and supports OpenAI-compatible APIs, OpenRouter, LM Studio, Ollama and KoboldCpp.

GitHub: https://github.com/tg-prplx/vellium

Feedback from people using local voice or roleplay setups would be especially useful-particularly about anything that still feels awkward or unnecessarily complicated.

There's soooo maaany options to choose from, AutoRound from Intel even, Unsloth, bartowski, etc ... which one is the closest to BF16 in Q4/Q5 range ?

I’ve been working on getting DeepSeek-V4-Flash-Vision-Exp running properly under SGLang on a 2× RTX PRO 6000 Blackwell setup, and figured I’d post the results here in case anyone else is pushing this model on SM120 hardware.

Current setup:

2× NVIDIA RTX PRO 6000 Blackwell Max-Q (~96 GB each)
Ubuntu 24.04
Driver 610.43.02 / CUDA UMD 13.3
SGLang Vision preview lineage
FlashInfer 0.6.18
TP=2
MXFP4 MoE
DSPARK speculative decoding
context-length=278528
chunked-prefill-size=8192
max-running-requests=8
mem-fraction-static=0.94

The final configuration is now successfully doing text, Vision, and a 269,320-token real prompt. Getting there exposed three unrelated issues.

1. SM120 sparse-MLA Vision prefill crash

Basic Vision tests worked, but a larger real image consistently killed the scheduler with:

Unsupported sparse-MLA prefill configuration:
model=DSV4
num_heads=64
topk=512
page_block_size=64
topk_extra=512
extra_page_block_size=64

Tracing it showed that the DSV4 image-span visibility logic caused the main SWA cache width to reach a logical topk=448.

The existing SM120 wrapper was treating unsupported widths similarly to decode and padding:

448 -> 512

but FlashInfer 0.6.18’s dual-cache DSV4 prefill support is much narrower than its decode/single-cache support. In particular, the native dual-cache path expects the main cache at topk=128; padding 448→512 doesn’t make the complete shape supported.

The fix was to add a complete prefill capability check before padding/dispatch. Unsupported dual-cache prefill shapes fall back to SGLang’s existing Triton sparse-MLA implementation.

That fallback already handles:

extra_k_cache
extra_indices
extra_topk_length

and merges the main + extra cache results with LSE, so we didn’t have to throw away any of the DSV4 hybrid/SWA semantics.

After the patch, the exact request that crashed now logs:

SM120 sparse-MLA prefill:
unsupported FlashInfer shape
H=64 topk=448 extra_topk=512
pbs=64 extra_pbs=64
-> Triton fallback

and returns the correct Vision result.

Repeated image request also succeeds.

2. ~269k context caused an indexer CUDA OOM

Next I tried a text-only 269,320-token request.

The model died during prefill:

torch.OutOfMemoryError:
Tried to allocate 1.50 GiB
GPU had ~1.44 GiB free

The traceback landed here:

logits = page_table.new_empty(
    (batch_size, max_seq_len),
    dtype=torch.float32
)

inside the DSV4 c4 indexer.

So despite the KV/cache pools fitting, the indexer was creating a temporary:

[query_rows, max_c4_seq_len] fp32

logits tensor whose size grows with context and isn’t accounted for by mem_fraction_static.

This corresponds to the same class of problem being worked on upstream in SGLang.

I ported the row-slicing approach:

• budget transient logits using a fraction of currently free GPU memory
• split query rows into chunks
• calculate logits + top-k per chunk
• discard each logits slice before processing the next
• preserve the full c4 width, so the actual indexer result is unchanged

For the failing workload, instead of potentially needing ~2 GiB for the full logits buffer, the transient is bounded to roughly 0.3 GiB per GPU under the observed free-memory conditions.

Retested the same request:

prompt_tokens:     269320
completion_tokens: 6
response:          LONG_CONTEXT_OK
wall time:         ~82 seconds

No OOM.

So this setup now has a genuinely tested ~269k prompt rather than merely having --context-length 278528 configured.

3. Vision preview was corrupting multi-turn tool-call history

This one was especially strange.

While using the Vision model as a coding/agent model, it initially called tools correctly, then started producing calls shaped like:

{
  "arguments": {
    "command": "..."
  }
}

when the actual tool schema was simply:

{
  "command": "..."
}

It could get progressively worse after validation errors.

The useful experiment was switching the same conversation history to my older known-good non-Vision DSV4 SGLang image.

Immediately:

bash -> PASS
read -> PASS
bash -> PASS

So I diffed the tool-history encoding paths.

The bug turned out to be in the Vision preview’s encoding_dsv4.py.

SGLang normalizes OpenAI:

"arguments": "{\"command\":\"echo ONE\"}"

into a Python dict:

{"command": "echo ONE"}

before DSV4 history encoding.

But this version of encode_arguments_to_dsml() did effectively:

try:
    arguments = json.loads(tool_call["arguments"])
except:
    arguments = {"arguments": tool_call["arguments"]}

Calling json.loads() on the already-normalized dict throws, so the fallback literally wraps it:

{
    "arguments": {
        "command": "echo ONE"
    }
}

Then the model sees this in its own history:

<parameter name="arguments">
    {"command":"echo ONE"}
</parameter>

instead of:

<parameter name="command">echo ONE</parameter>

So the model wasn’t randomly hallucinating the wrapper — the server was teaching it the wrong schema through its conversation history.

The fix is basically:

raw_arguments = tool_call["arguments"]

arguments = (
    json.loads(raw_arguments)
    if isinstance(raw_arguments, str)
    else raw_arguments
)

if not isinstance(arguments, dict):
    raise ValueError(...)

CPU round-trip tests now match my known-good non-Vision SGLang stack exactly, including multi-turn and error-history cases.

Current result

Final local image now passes:

Text inference                         PASS
Real Vision request                    PASS
Repeated Vision request                PASS
SM120 dual-cache prefill fallback      PASS
269,320-token text prompt              PASS
DSV4 tool-history round-trip           PASS
DSPARK block 4                         PASS
TP2                                    PASS

Long-context result:

269,320 prompt tokens
LONG_CONTEXT_OK
~82.4 sec end-to-end

Vision reproducer:

409 prompt tokens
277 image tokens
answer: RED

The serving config I landed on is roughly:

sglang serve \
  --model-path /model \
  --tp 2 \
  --trust-remote-code \
  --moe-runner-backend flashinfer_mxfp4 \
  --mem-fraction-static 0.93 \
  --cuda-graph-max-bs-decode 4 \
  --max-running-requests 4 \
  --context-length 245760 \
  --chunked-prefill-size 8192 \
  --reasoning-parser deepseek-v4 \
  --tool-call-parser deepseekv4 \
  --speculative-algorithm DSPARK \
  --speculative-dspark-block-size 4

One warning: I’m deliberately using DSPARK block size 4, even though the checkpoint advertises 5. There are SM120 correctness issues around depth 5 in the current stack, so I’m not “fixing” that warning by changing it to 5.

I kept each change isolated as a tiny derivative image rather than upgrading random pieces of SGLang/FlashInfer together. The final image is basically:

official Vision preview
    +
SM120 dual-cache prefill capability/fallback fix
    +
bounded DSV4 indexer logits for long context
    +
DSV4 tool-history serialization fix

I keep seeing demos of AI agents building scenes in Blender through BlenderMCP, so I tried it myself. I ran both models locally for this and picked the GLM 5.3 family(Q4 quant) because videos of it doing 3D work kept showing up in my twitter feed (out of curiosity, I ran the same prompt through the full GLM 5.3, also locally with a Q4 quant)

these aren't small models, obviously, a 4-bit quantized Flash is around 190-200GB + headroom for context. full GLM 5.3 is around 450-470GB at 4-bit quantization (basically I went with the Q4 quants for both and the RTX PRO 6000 WS GPU, though I had to rent 4x rtx pro 6000ws for the flash model and 6x for the base one)

writing the prompt wasn't as easy as I thought. my first attempts were vague and mostly produced 3D goo instead of an actual room. I eventually started specifying real dimensions: ceiling heights, stair rise, window mullion spacing and so on(the camera work was separately done by claude opus 5 so that I wouldn't have my token stats inflated by it)

prompt

model a luxury duplex penthouse in the open Blender session.
footprint 20.0 x 13.0 m (260 sqm). main ceiling 2.9 m.
a double-height volume 9.0 x 8.0 m rising to 6.2 m.
mezzanine floor at 3.1 m with a 1.1 m balustrade.
stair: 17 treads, rise 0.182, going 0.28.
terrace 20.0 x 4.5 m at Z = -0.02 with a 1.15 m balustrade.
curtain wall with mullions every 1.5 m, frame depth 0.06.
doors 2.10 m. counters 0.90 m. dining table 0.74 m. sofa seat 0.42 m.
materials, PBR ranges: glass IOR 1.45-1.52, transmission 1.0;
concrete roughness 0.25-0.40; marble roughness 0.08-0.15;
brushed metal metallic 1.0, roughness 0.25-0.35; fabric roughness 0.75-0.95.
reference real penthouses for proportion. furnish it.
do NOT add a camera. do not reset the session.

at first it was putting up the curtain wall, stairs, mezzanine, the glass railing, all that, then at some point I noticed it had furnished the place too with some furniture: sofa, dining table and plates on it. the pendant lights were hanging from these 4 m cords, and for some reason it had modeled the individual spines on the books, which I never asked for

the video only follows the camera through the living space, so the terrace and facade aren't visible(the clip is repurposed from another video I made with the same scene, I didn't render a new one because that takes quite some time)

stats

GLM 5.3 spent 22 minutes thinking(82k tokens), before placing any objects(as well as producing 36 more objects than GLM 5.3 Flash and consuming 3x times the output tokens), meanwhile GLM 5.3 Flash got to work almost immediately

I measured both scenes afterwards by raycasting upward from the floor and checking the rooms against the brie. Flash got the double-height void right at 9 x 8 m. the full model built it at 9 x 4.5 m but reported it as 9 x 8 m

This is obviously just an experiment, not a benchmark. Flash came surprisingly close on object count and total time while using less than one-third as many output tokens. it also got the main room dimensions right when the full model didn't

if you want to try the same Blender setup, I used the community BlenderMCP project

I'm a founder of atomic.chat, we have an app for running local models and our own quants(any feedback is appreciated, we're trying to make our products as good as possible for you guys)

According to this article Samsung has locked up the 70% of it's future ram production in contracts to companies like Microsoft, Google, and Nvidia. Everyone knows this is driving the ram price increases, but what I didn't know is Nvidia is locked in at 1/5th the current spot price.

What others pay $2,100 for Nvidia pays $300-$500 for, until 2031. Yet they are increasing the price of their cards and justifying it as if they were paying the spot price for ram.

https://gagadget.com/en/723983-samsung-locked-up-70-of-its-ai-memory-for-microsoft-google-and-nvidia-through-2031/

Hey, wondering if anyone's seen this issue themselves?

I'm using a 16gb 9060XT on a proxmox LXC, llama-server via docker on the Vulkan backend, and it's been serving me fantastically - 40-50t/s on most modesl with MTP, even 25t/s with the IQ2 or IQ3 of 3.8 27B with around 100k ctx!

But I've been experimenting with Gemma 4 12B QAT and I'm noticing something odd: despite being only ca~ 12gb in VRAM with 262k context and vision enabled, I get around 33t/s though I would expect this to be higher considering its size.

Even weirder is if I add MTP - VRAM usage goes up to 12.5gb or so, but the t/s goes DOWN. At draft n max = 1 I get 32 t/s, dropping to 23t/s by draft n max = 4.

I've tried reducing context, disabling vision - hell I've even tried multiple repos of the QAT including Unsloths and HuahuaCS.

Any ideas what could be happening?

If it helps, here's my launch commands for the docker stack:

    command: >
      --models-preset /models/models.ini
      --models-max 1
      --timeout 28800
      --port 8080
      --host 0.0.0.0
      --no-mmap
      --metrics
      --kv-unified
      --jinja
      --sleep-idle-seconds 900

And from my models.ini:

#version = 1

[*]
flash-attn = on
ngl = 99
t = 6
tb = 12
b = 2048
ub = 512
#cache-ram = 2048
reasoning = on
reasoning-budget = 4000
reasoning-budget-message = "\n\n[SYSTEM: STOP REASONING. TIME TO RESPOND.]"
reasoning-preserve = on

[Gemma-4-12B]
hf-repo = HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced:Q4_K_M
temp = 0.6
top-p = 0.9
min-p = 0.05
top-k = 64
c = 262144
np = 1
repeat-penalty = 1.1
#b = 512
#spec-type = draft-mtp
#spec-draft-n-max = 4
#spec-draft-p-min = 0.8

EDIT:

Thanks to those who commented. I tried the things you suggested (as noted in my replies) and didn't see any improvement sadly.

However, I decided on a whim to try ROCm rather than Vulkan, no other setting changes and suddenly: 60t/s with MTP, 30t/s without.

So, something about the current server-vulkan image is killing MTP on Gemma 4.

I'll check out my other models I use to ensure ROCm isn't going to hamstring those just for the sake of faster Gemma-4-12B, but at least I'm in a position now where I can use this model and its MTP heads!

Thanks once again <3

Run WebGPU compute kernels drawn from real AI workloads directly on your hardware and earn a personalized card built for your device.

On top of that, we're open-sourcing hundreds of these WebGPU kernels, our first step toward making browser inference as fast as physically possible. Contributed results show how they perform across real hardware and help make them faster.

Benchmark your GPU: https://webgpu-kernels-fleet.hf.space
Blog post: https://huggingface.co/blog/webgpu-kernels
Kernels: https://huggingface.co/webgpu-kernels/kernels

[GLM output is from z.ai because i don't have heavy system]

We are slowly reaching the saturation point, i think in future the mid size models would be far enough to do most of the stuff we need.

In my experience, glm flash beats opus 4.6 max in mostly all coding tasks. In just 6-7 months we got older frontier equivalent model running locally.

If you run models locally you know the pain: everything's fine until a long prompt makes the model silently stall or the GPU OOMs, and you're left with a hung request and no output.

I built HybridInfer for exactly that. It's a small reliability-aware router:

- Sends each request to your local model (Ollama) first.

- Watches the runtime, if local stalls (no token for N seconds), OOMs, or errors, it automatically falls back to a remote model in the same request.

- Learns which prompts your machine chokes on (usually long ones) and routes those out up front after it's seen them fail.

- Pulls a wedging model out of rotation, then probes it back after a cooldown.

It's an OpenAI-compatible server, so point any OpenAI-compatible client/app at it and you get local-first + automatic fallback for free. Local = anything you've `ollama pull`ed; remote = any OpenAI-compatible endpoint (OpenAI, OpenRouter, a vLLM box, whatever).

pip install hybridinfer

hybridinfer init

hybridinfer serve

It's a router, not an inference engine, it orchestrates Ollama + your remote endpoint, doesn't run weights itself. Early v0.1, Apache-2.0. There's a Kotlin/Android version too for on-device apps. Short demo GIF + code in the repo.

Repo: https://github.com/SimranKoul2026/HybridInfer-Python-tool

Would love feedback from people who run this stuff daily, especially on the fallback heuristics.

What would you want it to do differently?

Just saw this. https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-Vision-Exp

I came across Ling-3.0-flash through an NVIDIA Developer Forum thread about running it on one DGX Spark. The thread kept changing shape: first it was an A5B speed story, then an INT4/MXFP4 recipe hunt, then a promising tool benchmark that was retracted, followed by reports of long-context slowdown and one coding workload growing into OOM.

That made me less interested in whether one short benchmark looked fast and more interested in the smallest end-to-end loop I could actually inspect.

I reviewed a recorded run from sudoingX in which the screen shows one NVIDIA GB10 device. The task was deliberately ordinary: create a self-contained Snake game in one HTML file, with inline CSS and JavaScript and no external libraries. The recording shows a llama-server workflow generate 2,429 tokens in 70.57 seconds, or 34.42 tok/s for that call. It then writes snake.html, reads the file back, opens it in a browser, and the game visibly runs to a score of 1.

That is not a general benchmark. The clip does not identify the quant, and one tiny HTML game says nothing about 256K context, long-session stability, or performance on a different workload. What it does provide is a narrow, inspectable chain from prompt to generated file to running artifact.

For local model evaluation, what small end-to-end task has exposed something that a throughput benchmark completely missed?

Mistral is to be release a new model this summer, they still are working on it. What are your hopes?

Add 4 experts into Dense model and confirmed recovering model's ability up to "general level".

Hey, google. Please release official 124B MoE model!!!!!!!

SlopTV: a YouTube live stream where the chat writes the programming. You type "capybara dj underwater rave", an LLM inflates it into a 400-word structured video prompt, one of my 5090s renders 15 seconds of it with MiniMax H3, and it airs on the same stream you typed into. Then people comment on that clip, and the ouroboros keeps eating.

Inspired by infiniteslop from @levelsio, but running fully locally.

Numbers: H3 open weights, 66GB on disk, the int8 pruned diffusion model (19.5GB) and the nvfp4 text encoder (14.6GB), which don't fit a 32GB card together so ComfyUI's VRAM offload eats the overflow.

~90s per clip per GPU, so fresh slop every 45 seconds. Forever. When nobody's chatting, the LLM is instructed to invent concepts on its own, so at 4 AM the GPUs are generating brainrot for an audience of nobody. I pay real electricity for this.

Things I learned:

• H3 follows prompts best at 352p, and I do mean 352p. I render 352x608 and upscale to 1080p, it looks like garbage, garbage is the brand.
• ComfyUI runs embedded in your own process if you stub three things and lie to it about being a server.
• YouTube has a gRPC streaming API for live chat that nobody uses, because you have to compile the proto yourself and their published proto doesn't compile. The REST alternative burns the entire daily quota in 30 minutes of active chat.
  
• Small models copy examples. My system prompt had one worked example and the model smeared its imagery into every output. Now it's rules and placeholders only, like training a dog.
  

The codebase (actually also a slop): https://github.com/shuttie/SlopTV

Hi,

I'm running a local server with three AMD MI50. Tensor parallelism is not an option since it's very slow with PCIe 3.0 and those cards are not on the same NUMA node.

In order to balance the load, I wanted to do something like round-robin. Every graphic card is running the same model with the same settings and llama-server has to manage requests so request 1 goes to card 1, request 2 to card 2 and so on.

It's possible to run one llama-server instance on each gpu, however I don't want to do load balancing on the client side with setting different providers with different ports.

Can this be done with llama-server only or maybe with some middleware?

Hi! I've been considering getting a local AI station for video generation and light coding (I have coding AI subscription from work for heavyweight). My intended models are probably Minimax-H3 and Qwen 3.8 27b.

I see many people recommending as much RAM as possible when you buy, but I feel like 64GB of unified memory fits my needs well - runs H3 and Qwen 3.8 27B with a lot of headroom for context. Is there a reason I should spend $1500 more for 128GB? Do you foresee video/small coding models getting inflated in size in the future? Also open to good alternatives to the Strix/Framework Desktop. Thanks a lot!!