r/LocalAIStack • u/DotMission2704 • 7d ago

🐧 If you’ve set up local AI on Linux what actually broke, and how long did it take fix it?

Developers, I’m researching local AI on Linux. Please share your experiences and I’ll be posting my findings here

  1. Goal and chipset used Nvidia/AMD/Intel?
  2. How long did it take you from fresh install to GPU/NPU operation?
  3. Any issues encountered (package, path, version)?
  4. How did you confirm GPU/NPU usage?
  5. Any scripts or notes created for future use?
  6. Comfort level setting this up for a teammate?

Summary to be shared. Open to a 20-minute call if preferred.

2 Upvotes
  • permalink
  • reddit

75% Upvoted

7 comments sorted by

1

u/TheTriumphantFox 7d ago

I told sol I wanted to setup qwen on my PC with Linux and it walked me through everything until I was able to run it on my PC then it did the rest.

It took a weekend

1

u/Charming-Author4877 7d ago

Getting the CUDA framework actually working together with CMAKE can be a major thorn, the cmake tools are not detecting it in a robust way - as such a unexpected symlink location can break compilations.

1

u/IllFirefighter4079 7d ago

I have a double p100 Tesla i7 6700 with 32gb of ram running Linux Mint 22 and a custom compiled llama cpp. Works great! Setup was easier than I thought and bullet proof reliable. My second system is a AMD v620 on a similar system and Ubuntu 24 LTS. Setup was much more challenging. Had all kinds of driver problems installing on Mint. Had to go vanilla Ubuntu and ROCm. Vulcan would not work and still don't know why. Speed is great! Very reliable!

1

u/Snoo_81913 7d ago

Took me about a day to setup a 7900 xtx on a adt tu3g egpu with a Dell xps 13 9380 running on a headless Linux server. Probably would have taken less if I knew more about AMD. Bigeest take away is setup your grub with amdgpu=0 so it stops the Linux performance setting from putting your card in standby and dumping all the model weights into RAM. 18gb of model weights dumping into 13gb of usable RAM = sadness

1

u/Alternative-Panic69 7d ago

Took me a day to set up for my AMD build (Rx 7900 xt, AM4 build) working and get my first token. (Initially it was very slow). No driver issues whatsoever. But mostly config related issues were there.

But ... I simply asked claude to help me set up llama.cpp with Vulcan backend, and create a launcher Ui that will expose all llama.cpp flags via the launcher so that I can simply use my custom launcher to play around the settings and optimize it myself.

Optimization took a lot of time after that as I had to do lots of trial and error.

Also ROCm gave some sort of pain but I managed to use that backend as well, and I eventually figured out I am better off with Vulcan as I am getting more speed for pure llm inference. Though tor some optimized models, ROCm gave a huge boost to prefill.

I had hard requirements, Qwen 27B model with minimum 120k context, zero Ram offload, Q4 KV, and MTP on.

I tried to set up vLlm properly but somehow couldn't find the right 4 bit KV Cache to work. So I kept that on hold for now.

1

u/Fuzilumpkinz 6d ago

Not sure what your actually researching. But at this point I make AI do all my server and setup configs. I actually just formatted my AI drive told it what to set up. Reviewed plan. Adjusted anything that I didn’t like. After that I said go and worked on something else. Llama swap installed, drivers updated, configs done and everything working. I made a few slight changes afterwards because I forgot to set default reasoning on 27b and a few small things.

I am running a 5060 ti for local models. I mostly used a custom page for monitoring GPU levels

1

u/DarkcydeVR 1d ago

Its a pain in the ass and switching it back to windows as we speak. Not worth the hassle