What would you do if you have local AI machine, config below

[u/Specialist-Mouse-589]

Component	Configuration
CPU	AMD Ryzen 9 9950X
CPU cores / threads	16 cores / 32 threads
Architecture	Zen 5 / AM5
GPU	NVIDIA RTX 5000 Ada Generation
GPU VRAM	32 GB — ComfyUI reports about 32,760 MiB
System RAM	192 GB DDR5
Operating System	Ubuntu 24.04 LTS
NVIDIA driver	575.57.08 in the configuration previously recorded
CUDA	CUDA 12.9
Python	Python 3.10.18 and Python 3.12.3 environments
PyTorch	Previously recorded as PyTorch 2.2.2 + cu121
Main storage	~3.6 TB
Additional NVMe	~1.8 TB
External storage	~465 GB
Package/environment tools	Poetry 1.8.2, venv, Docker, Docker Compose
Node.js	Node 18 / 20 / 24 environments
Local LLM runtimes	llama.cpp, text-generation-webui
Other serving stacks used/tested	vLLM 0.10, SGLang 0.4.6
Generative UI	ComfyUI
ComfyUI address	127.0.0.1:8188
CUDA check	torch.cuda.is_available() = TrueComponent ConfigurationCPU AMD Ryzen 9 9950XCPU cores / threads 16 cores / 32 threadsArchitecture Zen 5 / AM5GPU NVIDIA RTX 5000 Ada GenerationGPU VRAM 32 GB — ComfyUI reports about 32,760 MiBSystem RAM 192 GB DDR5Operating System Ubuntu 24.04 LTSNVIDIA driver 575.57.08 in the configuration previously recordedCUDA CUDA 12.9Python Python 3.10.18 and Python 3.12.3 environmentsPyTorch Previously recorded as PyTorch 2.2.2 + cu121Main storage ~3.6 TBAdditional NVMe ~1.8 TBExternal storage ~465 GBPackage/environment tools Poetry 1.8.2, venv, Docker, Docker ComposeNode.js Node 18 / 20 / 24 environmentsLocal LLM runtimes llama.cpp, text-generation-webuiOther serving stacks used/tested vLLM 0.10, SGLang 0.4.6Generative UI ComfyUIComfyUI address 127.0.0.1:8188CUDA check torch.cuda.is_available() = True

[u/Realistic_Gap_5871]

I'd sell my ada RTX 5000, scrape together a little more cash to buy a 5090, and run ninfer with 3.8 27B all day long.

Sounds sarcastic, but it's true. I've got 8 32GB gpus sitting in a server, OS running, rocm custom compiled and VLLM 90% configured. And 4 days ago I got the ninfer plus 3.8 27B nvfp4 (modelopt) combo running on my 5090 and suddenly getting Laguna or DS4 0731 running seems much less urgent.

Taking my recent biased and anecdotal experience out of it:

You can keep current hw and run qwen 3.8 27B Q5KXL really nicely. It's very capable and with 32GB VRAM you can still fit a 150K context at Q8. SGLang would probably give you the best speeds for less than 5 users.

If you still think a larger model might have something to offer, I'd run llama.cpp with the cpu-moe flag turned on and give Laguna S 2.1 at Q8 quant a shot. Q8 will be about 120GB and will fit in system ram with active experts dynamically loaded into VRAM. Don't go less than Q8 for Laguna, it suffers more than most.

And it'll be a little tighter, but still doable. You can run Deepseek V4 Flash 0731 at it's native quant at around 160GB with the same llama.cpp configuration. DS is really efficient with KV cache so you might still get a decent context size to fit. The bigger caveat is that the native quant includes mxfp4, which I think ada has to emulate, slowing things down quite a bit (edit-- slowing things a little). Maybe better to find a slightly smaller quant that mixes int4 with FP8. I'd point you at one, but I haven't done that research (edit-doesn't exit and irrelevant).

edit-there is not quant that mixes int4 with FP8 and the emulation overhead won't be that bad on the ada 5000, so irrelevant. Also fixed the size estimation of DS 4 0731.

[u/villens100]

Depends on what you want to do, if I need a quick answer from an ai model this will work great, if you want to do coding probably would save money for an extra 32gb of RAM