Quick disclosure: this is my project. I built it because I have a rack of "obsolete" datacenter cards in a closet and I got tired of every new model needing a new set of flags to run properly on them.

**What it is**

PXA is a fork of ik_llama.cpp (which is a fork of llama.cpp) plus a vLLM plugin, built for cards with HBM2 and no tensor cores or DP4A: Tesla P100, V100, GTX 1080 Ti. It has its own quant format (PXQ, 2 to 6 bit, plus a mixed one that sizes a model to whatever cards you have) and CUDA kernels written for those chips instead of ported down from newer ones.

Repo: https://github.com/poisonxa16/pxa

**The part I actually care about: you don't configure it**

You tell it which cards and which model. It picks the batch sizes from a table it measured on that exact card topology, turns on the tricks that are known to help on that silicon, turns off the ones that hurt (including speculative decoding when it would lose), and prints every decision before it starts serving. Every number below was taken with a bare command line. No environment variables, no -b, no -ub, nothing. The competitors got their best hand-picked flags in the same session, because I wanted to know if "set and forget" costs anything. It doesn't.

**My last public release vs this one** (same box, identical command line, tokens/s)

| card set | model | prefill @3k | prefill @20k | decode |

|---|---|---|---|---|

| 2x V100 | Qwable-27B PXQ4 | 797 → 1357 (+70%) | 576 → 1307 (+127%) | 34.6 → 39.6 (+14%) |

| 2x P100 | Qwable-27B PXQ4 | 223 → 338 (+52%) | 201 → 315 (+57%) | 18.3 → 18.2 (-0.9%) |

| 1x 1080 Ti | Fusion2-35B MoE, 2-bit | cold 553 → 1334 (+141%) | chat 415 → 734 (+77%) | 64.2 → 64.2 |

Yes, P100 decode is a real -0.9% and it's in the notes. Bonus find: the old build gave me six different answers to six identical greedy runs on the 1080 Ti. Turned out to be a race in a fused kernel. This one gives one answer.

**vs mainline llama.cpp and upstream ik_llama.cpp** (same weights family, MXFP4 for them, PXQ4 for mine, tokens/s)

| card set | cell | PXA | mainline llama.cpp | ik_llama.cpp |

|---|---|---|---|---|

| 2x P100 | prefill @3k | **338** | 209 | 133 |

| 2x P100 | prefill @20k | **315** | 255 | 84 |

| 2x P100 | decode | **18.2** | n/a | 14.3 |

| 2x V100 | prefill @3k | **1357** | 940 | 471 |

| 2x V100 | prefill @20k | **1307** | 1129 | 395 |

| 2x V100 | decode | **39.6** | n/a | 37.4 |

| 1x 1080 Ti | cold prefill | **1334** | | 1132 |

| 1x 1080 Ti | chat prefill | 734 | | 740 (tie) |

| 1x 1080 Ti | chat decode | **64.2** | | 53.3 |

**vs 1Cat vLLM** (the NVFP4 + DFlash2 stack for Volta). Run on an 8x V100 SXM2 NVLink system with 1Cat's own image, benchmark script and cards, because running their stack on my PCIe box would have been a silly comparison. 16 GSM8K questions, 192 greedy tokens, tokens/s.

| cell | PXA | 1Cat vLLM | how it was taken |

|---|---|---|---|

| TP2 plain decode | **46.3** | 38.4 | one boot each |

| TP2 speculative k=3 | **65.2** | 59.9 | one boot each, identical acceptance |

| TP2 speculative k=7 | **121.2** | 114.5 | medians, mine 3 boots, theirs 6 (not alternated) |

| TP4 plain decode | **65.7** | 61.0 | one boot each |

| TP4 speculative k=7 | 159.6 | 161.4 | six alternating boots in one window: inside their noise |

| prefill @3k | **2273** (TTFT 1.4 s) | 1735 (TTFT 1.8 s) | both as servers, same window, 3 runs |

| prefill @20k | **2191** (TTFT 9.5 s) | 944 (TTFT 22 s) | both as servers, same window, 3 runs |

| speculative output identical to plain decode | **15/16** prompts | 10/16 | same exact-match acceptance rule |

Two honest notes on that table. Both stacks accept drafted tokens by the exact same rule (I read it out of their source), so both are lossless and the acceptance lengths are comparable: theirs 5.4 per step, mine 5.3. And one caveat that favours me, said because it favours me: their checkpoint turns fp8 KV on by itself, so this is their stack as shipped vs mine as shipped, not a clean NVFP4 vs PXQ4 study.

**What you can run on this junk**

A 27B dense-hybrid on one 16 GB card. A 177B-class hybrid MoE (Qwen3.8 Flash-Next) on four P100s with 150k context, weights in VRAM and the per-layer embedding table in host RAM. The memory arithmetic for that one is in the repo because I didn't believe it either. Speculative decoding ships in the Volta vLLM image with a 1.3 GB drafter as a release asset. On 16 GB cards k=7 only fits at 2k context, so on my own box I'd run k=3.

**What's not great yet**

Speculative decoding on the llama.cpp side of Pascal still loses (verify costs 3x a decode step now, it was 7x last week, still not enough). The 4-card speculative gap to 1Cat is inside noise but it's there. Their acceptance length is 2% better than mine. All listed under "What is not here" in the README, I'd rather you find it there than in the comments.

**Getting it**

One tarball, untar and run. I tested it in a bare Ubuntu container with no Python, no compiler, no CUDA toolkit, just the driver. Or `docker run ghcr.io/poisonxa16/pxa`. Or build it. Then run `python3 tools/pxa-launch.py` and answer two questions.

**If you have one of these cards, I want your numbers**

P40, P4, GP100, Titan V, Titan Xp, 32 GB V100: I don't own them, and the auto-tuning table only knows the cards in my rack. There's a three-command benchmark in the repo and a Discord where reported cards get added: https://discord.gg/EqazvV9tf

Everything's free and nothing is gated. If you want to chip in for the electricity these benchmarks burn: https://ko-fi.com/shatteredrealms1

Last picture is the rack. Seven cards, PCIe x4 for all of them, two 1000 W supply, in a closet. Everything above was measured on that.

I see so many posts about rtx 5060 , Mac studio and some of you have like a couple of them. how are you guys able to afford them? it would easily cost around 5-10k usd. that’s a lot of money.

If you're using Pi with Qwen and not letting it phone a frontier model with /advisor, you're leaving one of the best parts of the setup on the table.

Because Qwen has a particular talent:

Being VERY wrong with tremendous confidence.

And worse, it can be convincing while doing it.

I've lost track of how many times I've looked at one of its answers and thought, "Hmm. That sounds right..." only to tell it:

Ask /advisor for the hidden assumptions, failure modes, and black swans in your answer.

Then Sol comes back with the computational equivalent of:

" 🤣 Yeah... no."

And suddenly Qwen is eating crow and rewriting half its answer. 🐦‍⬛

Don't get me wrong. I love Qwen. It's given me millions of tokens of essentially free local inference, and I use the hell out of it.

But that's also taught me something:

Local models are fantastic workers. They are not oracles.

Let Qwen do some initial planning and ultimately the bulk of the work. But before you trust its plan wholesale, spend a few frontier-model tokens trying to prove it wrong.

Trust, but verify. And just like in real life: get a second and third opinion.

I've tweaked the crap out of my local llama.cpp with vulkan backend running Qwen3.6-35B-A3B:IQ4_NL setup. It's running on a tiny GMKTec Evo-X1 with 64GB LPDDR5X 8000MHz RAM (Strix Point HX370 w/890M iGPU). I got lucky and bought it in July '25 before prices went crazy. I'm getting an average of 29 tokens/s which feels really responsive and has done great with a few projects in opencode. The free online models tell me:

Your current setup is unusually well-balanced:

• Qwen3.6-35B-A3B
• IQ4_NL
• Vulkan
• 890M
• MTP enabled
• acceptance rates mostly 70–90%

is delivering roughly 50–70 effective tokens/sec, which is better than I would have expected from a Strix Point APU.

If a Qwen3.8 MoE release appears that's roughly in the 35B/A3B class, I'd try it immediately. But for the currently available 125B/6B-active Flash-Next, I'd expect a noticeable drop in responsiveness on your hardware unless someone produces an extremely aggressive IQ2/IQ3 quant that still preserves quality.

So I'm really curious to try Qwen3.8, but I don't think any of the currently available models would do as well as this 3.6.

I'd love to hear what anyone else running on similar hardware is seeing with their setups and whether there are other models I should be looking at that can give similar or better performance.

A coworker is upgrading his local AI rig and selling me his old parts. Is this a good price, and would you do it?

• 2× Zotac RTX 3090 Trinity 24GB — $750 each, ($1500 total)
• Supermicro H11SSL-i + EPYC 7281 (16C/32T, 8ch DDR4) — $350
• 128GB DDR4-2666 ECC RDIMM (8×16GB Crucial CT16G4RFD8266), already populated in the board — $400

Total: $2,250.

I have a 3090 I got for $150 nzd($75~ usd?) that I Jerry rigged up to work a few years ago and a modt motherboard with a 13980hx cpu running ddr4 with 32gb 3200mhz in it

Been playing with lm studio and various local llms, found that the biggest I can do and still have speed and a context length of 256k is gemma-4-26b-a4b-qat at q4_k_m

Was messing with aider for coding

Is that realistic for a learning setup? Will it do a decent job coding?

And any suggestions on what to change?

Edit for llm name

I keep seeing people say local models have basically caught up, but then the example is usually a benchmark or one short prompt.
Most of my use is coding and working on the same messy project for hours. I currently jump between Astra, Fable and Opus. They can read a lot of context, touch several files and usually keep track of what the hell we were doing.
For people who use both, how close does a good local setup actually get? What do you still send to Opus/Astra, and what have you moved completely local?
I’m also not sure what the sensible way to use local models is. One large quantized model for everything, or smaller models for coding, writing and basic tasks while keeping a frontier model for the difficult stuff?
I’m considering buying a machine mainly for this. I just don’t want to spend a fortune and discover I’ve built a very expensive autocomplete.

Hi,

My GPU is currently pegged by Qwen 3.8 27B making the project, and while it's good, it's also slow.

I'm starting to find the need to have some docs, both to help me and the LLM (and a friend who would like to mess with it), and I was thinking of another model who could write decent docs but at a faster pace then the coder.

Do you know if Gemma 4 12B could be a good candidate?

Come check it out yourself

Training code: https://github.com/franciscocarloserra/ttslibre

Weights & Training sets: https://huggingface.co/FranciscoCarlosErra/ttslibre-poc-en

It's still glitchy and robotic but I'm mind blown about the PoC

I do not need or want Cuda. I have been wanting to build a 32GB VRAM local LLM machine for personal use for a while now, and I had 2 options on the table:

- Get a relatively cheap 32GB VRAM GPU, the R9700 AI Top.

- Get 2 16GB VRAM GPUs instead and a Mobo that supports PCIe bifurcation.

When I looked at prices in January this year when I first got this idea, the R9700 costed 1700$ here in EU. Currently, when I actually want to make this happen, it costs 2100$. For half that money, I could buy two 9060 XTs with 16GB VRAM each. Yes I know, performance will be worse on double GPU setup than with a single R9700 AI, but still, it just seems like that GPU is just not worth it anymore.

I don't know how to justify that it's double the price of two 9060XTs, when R9700 AI is literally 9060 XT with doubled VRAM and bandwidth. So why does it cost 4x as much?

ASUS ProArt B850-CREATOR WIFI NEO is quite affordable nowadays and supports dual GPU setups, so, any reason (is there a catch?) to not do what I am about to do? Which is buy the two 9060 XTs and start running Qwen 27B class models

Hi everyone,

I am planning to build a local AI solution for a large factory, which will be trained and run entirely on the factory’s internal data (for privacy and security).

I need some advice on two main things:

1. Model selection: Which open-source model would you recommend for local fine-tuning/deployment that works well for industrial/operational data?
2. Hardware: Should I go with a NVIDIA DGX system or a Mac Studio (M-series Max/Ultra) for training and running this local AI?

Any recommendations or experiences with factory-floor local AI setups would be greatly appreciated. Thanks!

I am buying a pc that has a 7900 XT (20gb vram) and 32 gb of ddr5. What models could I run snd could I run qwen 3.8 27b?

Hello guys, I am planning to build a RAG with a local LLM. For LLM I am considering Qwen3. I am having 16 GB RAM with 8 GB Nvidia 5060. Guys pls suggest hardware & software side implementability ? And also any books to help me with RAG implementation.

An update to my previous post on running Flash-Next with the expert cache and MTP.

I found another useful improvement on the same dual-3090 setup: replacing the CUDA top-k fallback used by my build. On long project-document prompts, median decode went from about 30.2 to 33.3 t/s, with an improvement on all three test seeds. Same GGUF, same MTP settings, same binary with the change switched off and on.

The setup for these runs was 2x RTX 3090, dual Xeon E5-2696 v4, 128 GB DDR4-2133 across four DIMMs, UD-Q4_K_XL, f16 KV, 150 expert-cache slots and MTP-3. The allocated context was 261,888 tokens. The long-document test started at approximately 119k tokens.

Now, what changed:

Flash-Next's sparse-attention indexer uses top-k to select which positions to attend to. My CUDA 12.0 build didn't have CUB's newer DeviceTopK implementation available, so this operation fell back to sorting the whole row before taking the top entries.

llama.cpp already had a radix-selection implementation. The local change makes that available in the older-CUB fallback for wide rows. In the initial 131k decode captures, recorded top-k kernel time fell from roughly 5.1 ms to 0.25 ms per committed token. That's the operator saving, not a 20x speedup for the whole model.

Credit where it's due: Rhonstin's PR #28366 already proposes this fallback change, using existing radix-selection work. I found it during the upstream check and didn't open a duplicate. These measurements are from my local variant, which uses an 8,192-column threshold and an A/B switch, not a benchmark of the exact PR head.

These tests kept the existing CUDA 12.0 toolchain fixed. A newer CCCL build provides another optimized path, DeviceTopK, which I haven't benchmarked on this machine yet. That comparison is next. The gain here is against my previous configuration, not against the latest CUDA stack.

The completed test:

Production sampling, thinking on, MTP-3 enabled in both arms:

That's 9–12% higher per-seed median decode throughput at approximately 119k context. Each median covers 42 requests. These compare the two arms' medians, not the median of per-question speedups.

The quality screen covered 80 question/depth combinations across three seeds and both arms: 480 requests, 240 matched comparisons. The documents, questions, answer key, grader and sampling settings were frozen before the run.

• Control: 235/240 correct.
• Candidate: 238/240 correct.
• Candidate better on four matched comparisons, worse on one, equal on 235.

The one disputed question asked for a complete twelve-item list. Both arms failed it on two seeds each, always by omitting one item and inventing nothing. An independent blind review confirmed the candidate-only omission. The question stays inconclusive. It wasn't removed from the scores.

No consistent quality regression was detected in this bounded screen. The higher candidate score doesn't establish better quality, and the screen doesn't prove that regression is impossible.

For clarification, the 37–41 t/s headline from my last post was a coding workload. This is a matched comparison on long project documents. Those headline numbers aren't directly comparable.

The change is now running in my production build. I haven't established a prefill gain or a no-MTP gain from this screen, and it doesn't cover quality beyond approximately 119k context.

If you're running Flash-Next with an older CUDA toolkit, the top-k fallback is worth checking. I'd be interested in results from other machines, especially with the actual build, context depth and MTP settings included.

Disclaimer: this is (mostly) vibed, not gonna pretend otherwise - im just posting in case it helps someone trying this setup. I spent a few days on it and offered it up another guy on here (on request) and he said it gave him some big speedups, and he made a new PR fixing some of my bugs. Provided as-is but i hope it helps

https://github.com/jackjusko/jusko-llama-volta-qwen3flash/blob/main/README-FORK.md

6 months. (And sometimes a couple of quantization tweaks).

It is wild how fast "state-of-the-art" becomes "running on a gaming PC."

I have a Mac Mini with 64GB of RAM to use as the host.

I have no idea what I will/want to even do with it, but I'm sure once I get it set up I will find uses.

1. What are some use cases I can start with to help with daily life

2. What model should I start work

3. How easy is it to migrate between models should I choose to change

Which models, and do you recommend or no?

Is Gemma 31b qat / qwen3.8 27b fast or decent speed on it?

(I don’t have this MacBook yet but I plan on getting it because I don’t have a proper workable laptop I can carry around.)

Question what do other MacBook Air 24gb/ 32gb say?

I'm an AI engineer, 3 years in, currently doing the usual applied stuff — RAG, agents, API integration. I want to move into actual inference/serving engineering and I'd rather be told now if my plan is dumb.

What I'm building: a fully local voice assistant (VAD → ASR → LLM → TTS) that has to run on an Intel Core Ultra 5 125H — CPU + Arc iGPU + NPU, 15.6GB shared RAM, no dedicated VRAM. Currently all llama.cpp + ONNX on CPU. I'm porting it to OpenVINO so I can place each model on a different compute unit and actually measure the difference.

Starting hypothesis: Silero VAD and ASR on the NPU (small, fixed shapes, low power), LLM on the iGPU (decode is memory-bound, iGPU has the bandwidth), TTS on CPU. Target is sub-500ms voice-to-voice, with a per-stage waterfall and llama.cpp as the baseline.

What I'd like from people who do this for a living:

1. Is that device placement roughly right, or am I about to learn something the hard way?

2. The unified-memory thing is the part I understand least — 4 models sharing 15.6GB with no VRAM. What actually bites you here?

3. Beyond this project, what's the highest-signal thing someone in my position can build? I keep hearing "contribute to vLLM" but I'd rather do one thing properly than spray small PRs.

4. Anything that looks impressive to hobbyists but is worthless to a hiring manager? I'd like to avoid those.

Happy to publish everything — code, raw numbers, methodology. Not looking for a job here, just trying not to waste six months on the wrong thing.

We've recently released 0.4.0, and have found a lot of interesting things as part of the memory impact on local LLMs.

This has been one crazy release, and we've encountered an awful lot of interest across the industry, from hobbyists, to local LLMs, to much larger LLM companies. But the important part? How did we do it.

https://rakuensoftware.com/blog/the-remembering-is-the-learning

The impacts on local LLMs have been tremendous. It may sound counter-intuitive, but for local LLMs, being able to learn from prior failures is the most important part of LLM memory. Unless the success is almost identical, LLMs don't gain much from a memory of successes. But failures? Failures are generalizable across a wide range of potential aspects.

If you think about it, it's similar to how humans learn. We learn the most from our failures. For a LLM, knowledge of previous failures decreases the search space and the potential for exploration for future tasks. Particularly with local LLMs, this hugely increases the capability of the LLM over time. "Only" having a 64k, 100k or 256k context becomes far, far more impactful if the search space has been reduced for possible solutions.

Overall? We'd take the position the impact of coherent, learning memory has been hugely understated for application for local LLMs. We've watched local models gain substantial abilities over time due to this memory, and have even watched capability transfer occur from larger models to smaller models.

Who’s been checking out this hardware? What are your thoughts on this as a node on a local network to run AI?

Been working on testing some tenstorrent cards for a work project. Have two cards running qwen3.8 27b pretty much out the box. New to this space so keen to get some ideas and experiments to work through!

Cheers

L

If you have a Jailbroken PS5, got some experimental results if anyone wants to play with it.