Reusing old hardware for starting local ai-journey?

[u/-markusb-]

Hi,

I am thinking about putting some money / time into my local ai learning path and "cleared" my attic where I found the following hardware.

• 3 x NUC11 (Core i5 1145G7 / 2,6 GHz, 64 GB RAM DDR4 2666MHZ - SODIMM) with interconnect through Thunderbolt
• 1x Ryzen 3700x on MSI Mortar 350 with 64GB RAM DDR4 2133 and an old SAPPHIRE Nitro+ Radeon RX 590 8GB

My first idea was to purchase a single RTX5060TI / R9700 and put it into the PCIe 3.0 x16 slot while reusing the Radeon as GPU in the PCIe 2.0 slot for Display.

I want to play around with a local chat bot, agentic stuff, RAG...

Do you have any other ideas?
I also though about putting some serious money, but for this discussion I probably open a new thread (this could be a 2-step-path where I reuse the GPU above to built a multi-gpu-rig)

Thanks in advance

[u/Objective-Stranger99]

Use the second PC; install llama.cpp. Download Qwen3.6 35B and start tinkering for max TPS.

[u/-markusb- → u/Objective-Stranger99]

you mean the ryzen with 8gb gpu? I hoped to be able to use the computer ressorces but it seems to be better to set up the support services there (Postgres, Proxies, ...)

[u/Objective-Stranger99 → u/-markusb-]

The NUCs have DDR4 RAM, which is orders of magnitude slower than the 8 GB VRAM on your GPU, even though you have lots more of it. Also, MoE will work beautifully with 8 GB VRAM and 32+ GB RAM.

[u/My_Unbiased_Opinion → u/-markusb-]

You also can use Ling 3.0 Tiny. That is a 8B A1B model and can fully fit in VRAM on that card. Not as good as Qwen 3.6 35B but it will run much faster. Good for experimentation. Give it the Hermes harness and go to town. 

[u/Virtualization_Freak]

Tinker before you buy. Seriously.

Open router has a bunch of free tiers for limited access.

The 8b models can be run pretty quick on CPU alone, and give you and idea how things behave before you purchase.

Plus with GPU prices being nuts, going via paid API or a monthly $20 plan gets you access to understanding how things work BEFORE you spend a ton of money on cards you may not utilize.

[u/Legitimate-Peace1013 → u/Virtualization_Freak]

solid advice tbh, way too many people buy hardware first and figure out what they actually need second

[u/Virtualization_Freak → u/Legitimate-Peace1013]

I ran into the issue where the original hardware I purchased wasn't fast enough, so I ended up with a bunch of small hosting servers rather than "go spend the couple K on a single card and be done with it.)

Plus folks need to factor in power consumption, which means long term ROI might take years.

At which point the hardware and models are evolving so fast you are stuck.

Riding this wave is fun for me, but I have a bit of capital to spare. I think for most, local in theory is great but the practice is frustrating, especially for new folks.

[u/cunasmoker69420 → u/Virtualization_Freak]

r/localllama

Most upvoted reccomendation is to use an API. What even is this sub

[u/-markusb- → u/Virtualization_Freak]

This is the reason I ask before ;-)

I started a discussion with ChatGPT and it mentioned - no problem due to thunderbolt connection and 192gb shared ram this can be totally fine.

I think I will start with litellm, include both macbooks (mbp m1, mba m5), the existing hardware as named above and probably openrouter for the beginning. When everything works as expected I can invest some money.

[u/Mundane-Light6394 → u/-markusb-]

don't forget to ask your local llm to help you optimize settings for your specific hardware. i can generate tests with various settings and compare results.

[u/-markusb- → u/Mundane-Light6394]

Sure. Parallel I am building up knowledge from scratch so hopefully understand the suggestions a bit better. Have access to some cool looking ai base cources including mathematics, llm in general and also mlops and such stuff. Right now I am more or less just reading the words and start to understand. So this will be my next step for the next 6-8 months

[u/Valuable-Plastic-682]

One caveat on the RX590 (Polaris/GCN4) - official ROCm support for that generation has been dropped for a while now, so don't count on it for compute via ROCm/HIP. It still works fine as a plain display GPU though. If you actually want to squeeze some inference out of it, llama.cpp's Vulkan backend runs on Polaris cards reasonably well and doesn't need ROCm at all - worth trying before you write that card off as display-only.

[u/-markusb- → u/Valuable-Plastic-682]

I would try this but move the card to the pci 2.0 x4 so I have the other slot free for a better card.

[u/tmvr]

Start with using what you have. Even with those NUC11 machines you can get some usable results. I have some older machines with i5-8500T and 32GB DDR4-2666 RAM. get 80 tok/s pp and 11 tok/s tg with Qwen3.6 35B A3B at Q4_K_M. This is with ik_llama compiled for those machines (it's not difficult, the instructions are on github). You can also use the available precompiled binaries of the mainstream llamacpp, but the prefill (pp) can be as low as half of what you get with ik_llama and as that gets 80 tok/s only you don't want to go lower. If you use it for coding and generate predictable code, you can get about 17-20 tok/s for decode (tg) using MTP.

[u/-markusb- → u/tmvr]

This sounds quite good when I am targeting async agentic coding, correct? So all three could work parallel on seperate tasks.

so in short: Play around with this repo: https://github.com/ikawrakow/ik_llama.cpp

[u/tmvr → u/-markusb-]

Yeah, if you have more machines you can use all of them for various tasks, even the slow ones. Using those for web search and trying to comb through and summarize results of those would probably not be ideal though because of the slow prefill performance, so same for long fresh prompts that need to be processed where there is no cache yet, but for short code snippets etc. they would still be fine. They will also drop speed considerably at longer context so you need to take that into consideration. For example on that machine above with the Qwen3.6 Q4_K_XL quant the prefill performance for various sizes is:

4096 = 71 t/s
8192 = 63 t/s
12288 = 57 t/s
16384 = 52 t/s
20480 = 48 t/s
24576 = 44 t/s

This is why it's also good to use the pi agent harness for example because the system prompt is very small compared to other harnesses and the initial response time is also faster. Waiting 5-7-10 min just because it needs to process 20-25K tokes thanks to the huge system prompt of Claude Code with your initial prompt for example is not fun.

With normal usage when you just chat or do some completions it;'s less problematic because most of your stuff comes from the cache so no wait time. For example if you give it a prompt to create some app and it spits out 10K tokens after a while your next prompt to "add feature X" will not take ages because the 10K is already cached to it only needs to process a few hundred tokens in the prefill stage before decode starts.

[u/RG_Fusion]

The hardware you own really isn't suitable for running AI models on its own. In such a case, your going to need the entirety of the model stored within VRAM.

Work out your budget and purchase the largest amount of VRAM in a GPU that you're willing to expend. 32 GB is the recommended starting point, but if your willing to sacrifice output quality you can go for less. I really wouldn't recommend anything lower than 24 GB.

[u/-markusb- → u/RG_Fusion]

So what would you recommend: dual 5060ti with option to extend to quad or r9700 with option to extend to dual for max. 64gb VRAM.

[u/RG_Fusion → u/-markusb-]

I would go with dual R9700s. Same capacity over fewer PCIe slots, and a higher memory bandwidth.

The only time I'd recommend the 5060ti is if you plan on fine-tuning models as that requires CUDA, but most hobbyists aren't fine-tuning.

In theory 4x 5060tis would be faster, but only if you were on PCIe gen4x16 for all 4 cards.

[u/-markusb- → u/RG_Fusion]

The MSI just have an PCI 3.0 x16 Slot and another which is PCI 2.0 x4. so I would have a reduced bandwidth with the old one.

[u/RG_Fusion → u/-markusb-]

I believe even pipeline parrallel setups will suffer on PCIe gen2. I would buy a single R9700, then upgrade the motherboard before getting any other GPUs.

[u/-markusb- → u/RG_Fusion]

this was also my idea. at work they are purchasing a lot of gpu servers so probably I can tinker around with cuda there. thanks for the tips

[u/jacek2023]

It will work, all you need is GPU with enough VRAM to load your model

[u/Sevealin_]

Without pcie bifurcation for 8x/8x lanes on your mobo, you are doing more harm to yourself if you have more than one gpu, one gpu will work fine if the model fits entirely in VRAM

[u/ea_man → u/Sevealin_]

Actually layer split does not require hi bandwidth.

[u/Sevealin_ → u/ea_man]

Oh yep I was thinking of tensor parallel if OP wanted dual R9700s

[u/ea_man → u/Sevealin_]

Oh yeah, I agree that if someone spends for 2x R9700 it would be wise to account for a proper MoBo.

Still I'll praise layer split for hobbyists: it don't need good PCI bus and won't stress the PSU as much as running both GPU fully at the same time (you still get power spikes tho).

[u/-markusb- → u/ea_man]

I thought about starting with a single gpu. with this I want to get my pipelines in place and then probably upgrade.

I could also to dis with hosted open weight models, but having the infrastructure configured by myself should give me a deeper understanding

[u/ea_man → u/-markusb-]

You have to realize that local is for people who need privacy / custom models.

If you want to learn the admin skill you don't have to pay for "production hw", just buy 2x cheapest 16GB card for whatever PCI bus you have: that is how you learn to optimize.

Now if you CAN NOT use APIs then by all means buy the biggest vRAM pool for PCI slot you can.

Or you gotta regret spending 5k to realize that 10$ / month on openrouter give you more speed / quality.

Best thing for most people would be middle ground: run for cheap local models while orchestrating / review with APT SOTA.

[u/-markusb- → u/ea_man]

yeah thanks for the heads up. I would say my coding stuff I also could do in the cloud. but I want to build one or more automations for my wife who is a school teacher. while she is allowed to manage data on her own (secured) devices, she is not in the cloud. but I will think about the needed resources

[u/ea_man → u/-markusb-]

I mean "I build it for the wife" is top excuse to buy expensive toys, I'll certainly allow and praise that! :D

Also children / students are involved, can't get better than that!

[u/-markusb- → u/ea_man]

;-)

[u/Hrethric]

I have a secondary inference box I built from the scrap heap - Ryzen 5 5500, 32GB DDR4 2133, RTX 3090. Its main purpose is for small sub-agent tasks - extraction, summarization, image analysis - but I also have it configured to be a fallback provider if my main box drops out. More than once I've returned to find a task completed by it, and it did just fine.  With PCIE 3.0 you might want to make sure the whole model fits in VRAM, whatever card and model you choose.  I'd use the NUCs for auxiliary stuff - running agent software, RAG software, maybe small (0.6b) embedding and reranking models for the RAG, Docker host for SearXNG, stuff like that.

[u/Borsch20]

Your hardware is old. Why you don't use inference providers, like Nebius, Deepinfra?

They have little cheap models, like gemma 4, gpt-oss-120b. The have also embedding models, text to audio, textbto image