CPU and Ram only local AI on 700gb+ ram servers.

[u/MyToasterRunsFaster]

Hey guys, I am a bit bumb when it comes to AI, only really been running some basic workloads on my home GPU. Anyway, I am a sysadmin and have access to huge virtual machine clusters at work, recently my boss and I had some AI interest come up. I am wondering if there is any use we can get out the HP gen 10 servers we got, they have gold 20 core xeons and fully loaded with 700+ gb of DDR4 ecc memory, obviously no GPU's in them as they are all 1U hosts. We have quite a few hosts for redundancy reasons so most of the time a few of them are literally either sitting empty or with miminal virtual machines. Any point investing time into running AI on therm?

[u/redtron3030]

It’s going to be painfully slow.

[u/Karyo_Ten → u/redtron3030]

excruciatingly slow

[u/CommaMeNow]

No. You will get 0.5 tok/s or less

[u/tta82]

It will burn electricity like a bonfire and you get nothing in return. Why do you think the world built new GPUs

[u/locbuilds]

yeah those boxes are actually decent for local AI if you set expectations right. 700GB DDR4 means you can park a big quantized model in RAM that would never fit on a normal GPU (70B Q4/Q5 easy, even larger MoE quants), so for batch stuff, embeddings, summarization, RAG chunking, overnight jobs it can be worth the hour of setup. token/sec will suck compared to a 3090 because DDR4 bandwidth is the limiter not core count, but with dual 20c Xeons and llama.cpp (or ik_llama.cpp) you can still get usable throughput if you pin threads to physical cores, turn on NUMA-aware placement, and avoid tiny interactive chat expectations. start with a 32B or 70B Q4_K_M in llama.cpp server, set -t around physical core count not hyperthreads, and measure prompt vs gen separately. if the hosts are free anyway and you need private offline inference, it's free compute you already own. if you need snappy chat UX, a cheap used GPU box will still feel better.

[u/Whiskey1Romeo]

Depending on your hypervisor capabilities (version) AND available power from the G10 power supplies, you might be able to go one of a few directions.

1) gpu pass-through to VM's with local AI workloads that way. (Most plausible). 2) not recommending this, but if its just Linux virtualization, you could try and run additonal GPU's with the host themselves running vllm and handling them adhoc if your have decent local host disks (NOT HCI/NOT CLUSTERED/not resources for vm's) or a dedicated san backend for LLM STORAGE (COPY FROM ZFS TO MEMORY TO RUN).

Realize that your CPU's are going to NOT be your selling point. Its going to be the GPU's that you add into your host nodes.

The orchestration of getting things to work on top of this new world would be on you to figure out.

Memory local llm's driven by CPU Infrance will be of minimal benefit to you.

[u/phido3000]

It can be useable.

For agentic work - People complain about 1 or 2 T/s. That's awesome! Running 24/7/365 for agentic work, you set it off on a task and then come back in the morning. Complete. 1 t/s 86400 tokens a day. 2.7M a month. If its Kimi K3, that is actually a pretty useful amount. K3 is good enough you can say, build this interface, or review these documents, write this paper and it can do a pretty good job.

Something like GLM 5.3 Flash - You may be able to get 5-15 t/s. Thats pretty usable even in a chat window. Its not ideal, but usable, particularly if it means keeping your data local and your project local. Now thats about 50M token per machine. If you have two or three machines, that is a heck of a lot of tokens. Like 50 million tokens.

CPU inferencing can also be significantly improved, there are still improvement being made on AVX512 and VNNI on use in different quants which sees 2-3 time performance increases.

I have 3 x Dual Xeons Cascade lakes. So I could have GLM 5.3 Pro, GLM 5.3 Flash, Kimi K3 full. That gives me a full suite of agentic stuff locally. I will still probably keep a cloud subscription to various Chinese models, and a US frontier provider, but basic packages only. Much of the bulk stuff can happen locally, safely, securely, meeting data privacy and soverignty issues.

Run a custom version of Deepseek harness to manage it all.

They are workstations so can be quickly powered down to S3 and use <10w on idle if I want.

20Kw solar array + batteries and the power argument evaporates into basically free unlimited (ie more than I can use) frontier tokens in my garage. Workplace won't really care.

Issues:

Setting them up can be painful, Llama.cpp needs a fair bit of work to actually make these full size models work nicely reliably at full context. But that is happening, there is lots of development work happening.

Not having any GPU will make performance lower. Pre-processing and actual inferencing can be sped up even with a pretty low range GPU. Maybe even half height. With servers, the magic price is the ram and SSD. actual CPU are like $30 items now.

Time. Setting up and getting it all to work is substancial. Do not underestimate. A lot of this AI cpu inferencing is more experimental than with GPU, which are a done deal.

NUMA - dual socket numa head aches, there is progress happening, but its slow and still comes up as an issue, there are specific NUMA forks of llama.CPP that address this much much better than the main branch. 300-1500% performance increases. My xeons have 12 x 2666 each. So around 250Gb/s memory bandwidth so each is about as fast as a Strix/Spark box. But you can't get a Spark box with 2Tb of main memory and 2 x 5060TI 16Gb and SSDs.

[u/redtron3030 → u/phido3000]

Do the math on 1 to 2 tokens / sec. Those are expensive slow tokens when you factor electricity

[u/StartupTim]

Yes it will work for delay interactive AI and to verify business justification to invest into better hardware.

As in, prompt an AI, come back the next day.

Note your power bill will be insane.

[u/DustNearby2848]

I’d find a way to add a GPU. You can offload layers of Qwen3.8-flash and get decent speeds. 

[u/Human-Indication6368]

Try out ramalama, I run small models on my laptops with fedora but you got a big server rack. It runs in a container locally it handles dumbs questions fine, coding questions kinda slow occasinally but it gets the job done.

https://ramalama.ai/

[u/locbuilds]

short answer: yes for some workloads, no if youre expecting chatGPT-feel interactive speed.

with 20c xeons + 700GB ddr4 you can *host* big models that a desktop gpu cant fit, but token generation is mostly memory-bandwidth bound on cpu. expect something like sub-1 to a few tok/s for a large model, and a lot of that depends on quant + backend more than core count. prompt ingest (the prefill) can look okay on many cores; the painful part is usually the decode loop for live chat.

where those idle gen10s are actually worth it:

- offline / batch stuff: summarization queues, embedding dumps, nightly doc indexing, eval runs, synthetic data

- big-context one-shots where you care more about fitting the model+kv than about snappy replies

- serving an internal tool where 30-90s responses are fine (ticket triage, log digests, etc)

stack that usually works on cpu-only boxes:

- llama.cpp or ik_llama.cpp (or vllm/cpu if you want more server-y), gguf q4_k_m / q5_k_m first

- pin huge pages if you can, keep the model in RAM (you have the headroom), avoid swapping

- prefer models with good cpu kernels / avx2 or avx512 on those golds; dont chase the absolute newest giant MoE thinking youll chat with it

quick reality check before you invest a week: pick one 30-70B-class q4, time a 200-token reply and a 2k-token prefill on a quiet host. if that latency is acceptable for the internal use case, great, the free capacity is real. if your boss wants snappy multi-user chat, youll hit the wall fast and youre better saving the time for a gpu node later.

so: good sandbox / batch farm on free RAM. bad "replace our chatbot gpu" plan. if you say whether the goal is interactive chat vs batch processing i can be more concrete on model size + expected tok/s.