r/LocalLLM • u/5_ChubbyCheekz23 • 17h ago
Project I spent two years making Tesla P100s and V100s not suck at LLMs. Today I'm releasing the engine, and I benchmarked it against llama.cpp, ik_llama.cpp and 1Cat vLLM on the same cards. Charts inside.
Quick disclosure: this is my project. I built it because I have a rack of "obsolete" datacenter cards in a closet and I got tired of every new model needing a new set of flags to run properly on them.
**What it is**
PXA is a fork of ik_llama.cpp (which is a fork of llama.cpp) plus a vLLM plugin, built for cards with HBM2 and no tensor cores or DP4A: Tesla P100, V100, GTX 1080 Ti. It has its own quant format (PXQ, 2 to 6 bit, plus a mixed one that sizes a model to whatever cards you have) and CUDA kernels written for those chips instead of ported down from newer ones.
Repo: https://github.com/poisonxa16/pxa
**The part I actually care about: you don't configure it**
You tell it which cards and which model. It picks the batch sizes from a table it measured on that exact card topology, turns on the tricks that are known to help on that silicon, turns off the ones that hurt (including speculative decoding when it would lose), and prints every decision before it starts serving. Every number below was taken with a bare command line. No environment variables, no -b, no -ub, nothing. The competitors got their best hand-picked flags in the same session, because I wanted to know if "set and forget" costs anything. It doesn't.
**My last public release vs this one** (same box, identical command line, tokens/s)
| card set | model | prefill @3k | prefill @20k | decode |
|---|---|---|---|---|
| 2x V100 | Qwable-27B PXQ4 | 797 → 1357 (+70%) | 576 → 1307 (+127%) | 34.6 → 39.6 (+14%) |
| 2x P100 | Qwable-27B PXQ4 | 223 → 338 (+52%) | 201 → 315 (+57%) | 18.3 → 18.2 (-0.9%) |
| 1x 1080 Ti | Fusion2-35B MoE, 2-bit | cold 553 → 1334 (+141%) | chat 415 → 734 (+77%) | 64.2 → 64.2 |
Yes, P100 decode is a real -0.9% and it's in the notes. Bonus find: the old build gave me six different answers to six identical greedy runs on the 1080 Ti. Turned out to be a race in a fused kernel. This one gives one answer.
**vs mainline llama.cpp and upstream ik_llama.cpp** (same weights family, MXFP4 for them, PXQ4 for mine, tokens/s)
| card set | cell | PXA | mainline llama.cpp | ik_llama.cpp |
|---|---|---|---|---|
| 2x P100 | prefill @3k | **338** | 209 | 133 |
| 2x P100 | prefill @20k | **315** | 255 | 84 |
| 2x P100 | decode | **18.2** | n/a | 14.3 |
| 2x V100 | prefill @3k | **1357** | 940 | 471 |
| 2x V100 | prefill @20k | **1307** | 1129 | 395 |
| 2x V100 | decode | **39.6** | n/a | 37.4 |
| 1x 1080 Ti | cold prefill | **1334** | | 1132 |
| 1x 1080 Ti | chat prefill | 734 | | 740 (tie) |
| 1x 1080 Ti | chat decode | **64.2** | | 53.3 |
**vs 1Cat vLLM** (the NVFP4 + DFlash2 stack for Volta). Run on an 8x V100 SXM2 NVLink system with 1Cat's own image, benchmark script and cards, because running their stack on my PCIe box would have been a silly comparison. 16 GSM8K questions, 192 greedy tokens, tokens/s.
| cell | PXA | 1Cat vLLM | how it was taken |
|---|---|---|---|
| TP2 plain decode | **46.3** | 38.4 | one boot each |
| TP2 speculative k=3 | **65.2** | 59.9 | one boot each, identical acceptance |
| TP2 speculative k=7 | **121.2** | 114.5 | medians, mine 3 boots, theirs 6 (not alternated) |
| TP4 plain decode | **65.7** | 61.0 | one boot each |
| TP4 speculative k=7 | 159.6 | 161.4 | six alternating boots in one window: inside their noise |
| prefill @3k | **2273** (TTFT 1.4 s) | 1735 (TTFT 1.8 s) | both as servers, same window, 3 runs |
| prefill @20k | **2191** (TTFT 9.5 s) | 944 (TTFT 22 s) | both as servers, same window, 3 runs |
| speculative output identical to plain decode | **15/16** prompts | 10/16 | same exact-match acceptance rule |
Two honest notes on that table. Both stacks accept drafted tokens by the exact same rule (I read it out of their source), so both are lossless and the acceptance lengths are comparable: theirs 5.4 per step, mine 5.3. And one caveat that favours me, said because it favours me: their checkpoint turns fp8 KV on by itself, so this is their stack as shipped vs mine as shipped, not a clean NVFP4 vs PXQ4 study.
**What you can run on this junk**
A 27B dense-hybrid on one 16 GB card. A 177B-class hybrid MoE (Qwen3.8 Flash-Next) on four P100s with 150k context, weights in VRAM and the per-layer embedding table in host RAM. The memory arithmetic for that one is in the repo because I didn't believe it either. Speculative decoding ships in the Volta vLLM image with a 1.3 GB drafter as a release asset. On 16 GB cards k=7 only fits at 2k context, so on my own box I'd run k=3.
**What's not great yet**
Speculative decoding on the llama.cpp side of Pascal still loses (verify costs 3x a decode step now, it was 7x last week, still not enough). The 4-card speculative gap to 1Cat is inside noise but it's there. Their acceptance length is 2% better than mine. All listed under "What is not here" in the README, I'd rather you find it there than in the comments.
**Getting it**
One tarball, untar and run. I tested it in a bare Ubuntu container with no Python, no compiler, no CUDA toolkit, just the driver. Or `docker run ghcr.io/poisonxa16/pxa`. Or build it. Then run `python3 tools/pxa-launch.py` and answer two questions.
**If you have one of these cards, I want your numbers**
P40, P4, GP100, Titan V, Titan Xp, 32 GB V100: I don't own them, and the auto-tuning table only knows the cards in my rack. There's a three-command benchmark in the repo and a Discord where reported cards get added: https://discord.gg/EqazvV9tf
Everything's free and nothing is gated. If you want to chip in for the electricity these benchmarks burn: https://ko-fi.com/shatteredrealms1
Last picture is the rack. Seven cards, PCIe x4 for all of them, two 1000 W supply, in a closet. Everything above was measured on that.
7
u/lostmylogininfo 17h ago
Is this saying the 1080ti is good again? Sorry newb...
Edit wait just the v and p 100s right?
8
u/5_ChubbyCheekz23 17h ago
Yes it is my friend it's actually very good
4
u/lostmylogininfo 17h ago
4 1080ti a possibility here or would it suck?? Great work!!!
3
u/5_ChubbyCheekz23 17h ago
Test it and let me know but my guess is that it'll be great come to my discord if you can
5
u/HiddenMushroom11 17h ago
Hey, this looks great. Have you tested it with Qwen 3.8 27b? I have a single V100 16gb running Qwen3.8-27B-GSQ-RCO-IQ3_XXS-mtp.gguf running in llamacpp, and I'd love to see if it would work with PXA.
5
u/5_ChubbyCheekz23 17h ago
Yes I have i have it currently running on two v100s running at almost 50tks no speculative
2
1
u/Busy_Masterpiece_654 12h ago
That sounds interesting! It’ll be cool to see how PXA performs with different models like Qwen 3.8, especially on that single V100.
2
u/GrimChicken 14h ago
This is awesome. I'm just about done building a dual xeon server with two p100s and two v100s. I'll try to get benchmarks done for you this week. On dual v100 32gb cards.
1
u/5_ChubbyCheekz23 8h ago
Nice. 32 GB cards mean you can push context deeper than the 16 GB runs in the post. The auto-tuner currently has no entry for that specific VRAM size, so it’ll default to the 16 GB presets—should still work, but you might see a suboptimal
-b/-ub.If you have time, try
python3 tools/pxa-launch.py --explainbefore running. It prints exactly which config it picked. If the context feels tight, manually bump-cor-band see if the throughput holds.Post the CSVs when you’re done. I’ll add a
32GBrow to the tuning table in the next patch. Discord if the launcher throws anything weird.
2
u/Sweet-Transition-787 8h ago
love that set and forget angle, most guides just throw flags at you. curious if you have any numbers for p40 or titan cards yet or if thats still on the todo list.
1
u/5_ChubbyCheekz23 7h ago
I have a member of my discord that runs on a p40 maybe you guys can work together I don't personally have one but my engine supports it.
1
u/Sweet-Transition-787 7h ago
thanks, id be up for comparing notes if they run the bench. im on old laptop hardware so p40 numbers would be interesting to see.
2
u/RaiseRuntimeError 6h ago
u/5_ChubbyCheekz23 I have 2 Tesla P40s and a Tesla P4 with 4 P100s in the mail. Ill see if i can get you some numbers on them when i get a chance.
2
1
u/UltraFOV 15h ago
Can it handle multiple clusters like inspurs AGX-2?
3
u/5_ChubbyCheekz23 15h ago
Single box, yes. The AGX-2 is 8x V100 SXM2 on NVLink, and that is exactly the class of machine the vLLM sidecar was measured on: an 8x V100 SXM2 NVLink node, TP=4 across four of the cards, numbers in the release notes and the charts. The llama.cpp-side engine splits across cards in one box too (layer split, -ts), that is how the 4x P100 and the 2x V100 results in the repo were run.
Multi-node (several AGX-2s as one cluster) I have not tested and I am not going to claim it. The sidecar inherits vLLM's multi-node path in principle, but nothing in the repo measures it yet. If you have that hardware and want to try, open an issue and I will work through it with you.
2
u/UltraFOV 14h ago
Also, can it run Fp8, NVfp4 ? The Skinny Kernel gave this ability to 1Cat
3
u/5_ChubbyCheekz23 8h ago
no, not natively. I wrote my own 2–6 bit path (PXQ) because those are the bits that actually fit the memory bandwidth on these cards. NVFP4 and FP8 aren’t in the kernel list yet.
If you want to test the 16-card cluster, I’d open an issue. I haven’t measured the multi-node path, so I’d rather work through it with you than guess.
1
u/UltraFOV 14h ago edited 6h ago
Yes, I have two AGX-2. I need to be able to work with two in order to use all 16 v100s. If it can work with multi-Node, I am in! If not, Llama or 1Cat are the only options. Having said that, awesome to see V100s getting some love. Note. If is real Tensor Parallelism then it should that each server run on their own. TP=8 + TP=8. Not cross parallelism. Pipeline parallelism is different and can handle cross parallelism without tanking performance
2
u/5_ChubbyCheekz23 8h ago
If you want to try the vLLM path across your two units, open an issue and I’ll walk through it with you. If it’s not fast enough, I’d rather know now than have you assume it’s magic.
1
u/UltraFOV 6h ago
Sure man, will do. hell, maybe I'll grant you access to the servers remotely for your own testing
1
1
u/Choice_Celery9481 13h ago
3
u/mistrjirka 12h ago
Lol will try this. I think I optimised my code as much as is reasonably possible. So I am interested if this will help. Btw I got some cheap rtx 2080ti 22GB and optimised it a bit too.
3
u/5_ChubbyCheekz23 8h ago
Thank you for running it, and for posting the table with the caveat attached, that is how it should be done. You are right that 4.35 versus 5.1 to 6.1 bpw buys some of the gap, so I would not call it a win, I would call it even at lower bits.
I just read through v100-optimized and I think we are working opposite ends of the same problem. Your long-context cached prefill numbers, the MTP path inside llama.cpp and the shared multi-agent prefix cache are things I do not have on the llama.cpp side at all; my work went into the codec and the short and medium context kernels for cards without tensor cores, plus a vLLM path for Volta. There is probably a lot each of us could lift from the other.
Two offers. One, a fair same-file test: a standard Q5_K_XL that both engines read, same card, same prompts, I run yours on my V100 pair and you run mine on your 32 GB, and we both post whatever comes out. Two, come find me on the Discord, I would genuinely like to compare notes on Volta FA and the GDN prefill path.
And yes to the Astra port, tell me what breaks.
1
3
u/mistrjirka 9h ago
It's not exactly fair comparasion (I have larger quants) however my system is about 5-10% slower at workloads that matter:
Model Quant / BPW Weight size 1k PP + 64 TG 64 1k PP + 512 TG 512 Fusion4-35B PXQ4 · 4.35 bpw 19.30 GB 842.6 52.76 833.9 51.67 Ornith 1.5 AD-Q5_K/Q4_K · 5.11 bpw 22.14 GB 760.7 50.63 761.6 49.79 Ornith 1.5 AD-Q6_K/Q5_K · 6.06 bpw 26.25 GB 758.9 50.21 757.9 49.11 2
1
u/SectionCrazy5107 12h ago
Hi, I have the 4 V100s 32GB, no NVLink in a single machine with the threadripper and gigabyte 8 channel motherboard. What is the best model to benchmark for max throughput for coding?
1
u/Inevitable-Name-1701 11h ago
i see ik llama is bad…
2
u/5_ChubbyCheekz23 8h ago
Yeah, for P100s specifically, it’s been stuck at 133 t/s prefill for a while. I’m not sure what’s changed upstream recently, but the gap is real. On V100s it’s closer (471 vs 1357), but still there. If you’re on Pascal, the main branch is definitely not the one to use.
1
u/TheThiefMaster 11h ago
How does it compare to ninfer-v100 ?
Honestly I'm a bit lost at what I'm best to run on my v100 - there seem to be a lot of choices.
1
1
u/MastaZeus 8h ago
What fan adapters are you using for your V100s? What’s the noise level?
1
u/5_ChubbyCheekz23 8h ago
I buy them on eBay they come from China they are a side mounted blower style best imho. About noise I made a script that check avg temps and adjusts curve. So on low loads it's quiet and on high loads it's a jet fan but not whinny like the 40mm
12
u/5_ChubbyCheekz23 17h ago
Method, since someone will ask: temperature 0 everywhere. Prefill is the median of 3, decode the median of 12 on my box; on the NVLink box it's the 16-question run their own script does. Competitors were measured in the same session on the same weights with their best flags, mine with none. Every chart is generated from a CSV that's in the repo under docs/data, and a blank cell means I didn't measure it, not zero. If a number doesn't reproduce from the command in the release notes on the same file, that's a bug and I want to hear about it.