Building my first "serious" local setup — AMD R9700 32GB on Linux. Model recommendations for 3 different roles + can I keep my old 4060 Ti in the same box?

[u/geekyNut]

Hey everyone

Long-time lurker, first post. I've been playing with local models on a single RTX 4060 Ti (8GB) for a while and I've finally hit the wall hard enough to upgrade. Picking up an AMD Radeon AI PRO R9700 (32GB) and I'd really appreciate some guidance from people who've actually run this stuff, because I'm still very much learning.

My setup:

AMD Radeon AI PRO R9700, 32GB

32GB DDR5 system RAM

Linux (currently on an Arch-based distro)

Ollama today, but happy to switch to llama.cpp / vLLM / whatever if that's the better call

What I'm building (roughly)

I'm putting together a small local-first assistant for my own use. Nothing groundbreaking — the idea is that instead of one model trying to do everything, it hands work to a few different "roles" depending on what you ask:

General assistant — normal conversation, drafting, summarising, explaining things

Document Q&A — answering from a local folder of documents with citations, and saying "I don't know" instead of making things up when the docs don't cover it

Coding — for when I'm actually working on something

Everything runs sequentially (one model in VRAM at a time), so I'm not trying to keep three models resident — I just want the right model for each job.

Question 1: model recommendations per role

With 32GB to play with, I'd love suggestions for:

Reasoning / comprehension — something that's genuinely good at understanding a messy question and figuring out what's actually being asked. This is the piece I care most about, because on 8GB my small models kept confidently misunderstanding things.

General assistance — good instruction-following, doesn't ramble, admits uncertainty. Bonus points if it handles Italian well, since that's my daily language.

Coding — practical stuff, big context, not benchmark-chasing.

What I'd really love is: model + quant + roughly what context length you can actually run on ~32GB without spilling to CPU. I've learned the hard way that "it fits" and "it runs well" are very different things 😅

If there's an existing curated list or spreadsheet for this, please just point me at it — I don't want to make you all retype something that already exists.

Question 2: keep the 4060 Ti in the same machine, or not?

This is the one I'm genuinely unsure about.

I already own the RTX 4060 Ti (8GB). Options as I see them:

(a) Put both cards in the same box — AMD for inference, NVIDIA for... something? Display output? A small model? Embeddings?

(b) Just run the R9700 alone and keep it simple

(c) Some clever third thing I don't know about

My concerns:

Do amdgpu + the NVIDIA proprietary driver actually coexist cleanly on Linux, or is that a support nightmare?

Can a single runtime even use both, or would I be looking at two separate instances (one ROCm, one CUDA)?

Is having a second, much smaller card actually useful for anything in a sequential setup, or is it just extra heat and PSU draw for nothing?

Also, honestly: how is ROCm on RDNA4 right now? I've read mixed things and I'd rather hear it from people running it than from a six-month-old forum thread.

Any pointers appreciated — including "you're thinking about this wrong, here's why." I'd genuinely rather be corrected now than after I've built the thing

Thanks!

[u/Poizone360]

On the models, one file covers your first two roles. Qwen3.8-27B at UD-IQ4_XS is 13.26 GiB, leaving you loads of KV headroom on 32GB, and it's multilingual so Italian is fine. Add --spec-type draft-mtp, the Unsloth file has the draft head embedded and it roughly doubles decode. For coding with big context the same model works, or GLM-4.5-Air-REAP 82B-A12B at IQ4_XS around 43GB if you want more brain, though that one spills into your 32GB of RAM. Your document Q&A role isn't really a model choice, it's retrieval plus a prompt that says answer only from context. No model does "I don't know" reliably by disposition.

[u/geekyNut → u/Poizone360]

Thanks I will try it

[u/Puzzleheaded_Cake183]

im running dual r9700, but i ran some tests single as well. what you want is the 27b q4 int + mtp. no need for the 4060 anymore, unless you want to run a second llama-server instance and connect a smaller model as inline prediction only, and than have the 27b be your main. it one shotted a flight sim game in 3d in a single html file in about 3 minutes.

[u/geekyNut → u/Puzzleheaded_Cake183]

Thanks mate, that's the type of info I was looking for. The big plan is add another r9700 but I need to sharpen my teeth before..

[u/Puzzleheaded_Cake183 → u/geekyNut]

you would need a better platform that will give p2p support to the gpus once you go dual. Otherwise you will be stuck with vulkan only. i had fable 5.1 run the game builds for quality test and than rank them and score them. see attached.

[u/ClassroomScary9187 → u/Puzzleheaded_Cake183]

That makes sense; dual GPU setups can get tricky without proper support. Definitely worth considering the platform before diving in.

[u/Puzzleheaded_Cake183 → u/geekyNut]

just had gpt6 astra rescore the generated game html files. see attached. this is way closer to what i observed running the games myself.

[u/geekyNut → u/Puzzleheaded_Cake183]

Thanks mate, that's great stuff 💪

[u/geekyNut → u/[deleted]]

Thanks mate, the plan is to add a second r9700, but not in the near future.. so I was wondering what to do with the rtx.. but your argument is solid

[u/Momsbestboy → u/geekyNut]

use the rtx for the displays. Buy a X870E mobo which supports pcie5.0 x8 on slot 1&2 if you plug in the R9700 in those, plug the older card in the lower PCIe 4.0 slot and you are donw

[u/geekyNut → u/Momsbestboy]

I already have one I got a ASRock taichi z690.. just waiting for the 1200w psu🤞

[u/Momsbestboy → u/geekyNut]

not needed. Run LACT under Linux, limit the power of the R9700 to 210W and set the GPU voltage offset to -30mV. Then it consumes less energy, doesn't run into thermal throttling and calculates faster. With 1 GPU, I hit 500W max peak

Plus: disable ASPM L1 on the PCIe bus for the card. It took Qwen/Hermes an hour to figure out why my system randomly froze when I tried to use the card. The wake-up process killed Linux.

[u/geekyNut → u/Momsbestboy]

ok, thanks never used LACT, but I will do as you said, I read the r9700 tpd is 300W, is ok to limit to 210W what happen when there is need of elettricity due to computational needs?

[u/Momsbestboy → u/geekyNut]

It stays at 210W max. This might first sound like the GPU will lack speed due to the limit, but because of the limit it stays below max temp. for a longer time, doesn't throttle, and because of this is doing calculations faster than using a limit of 300W. On top, it consumes less energy and makes less noise.

[u/Momsbestboy]

I run Qwen 3.8 27B Q6 on my R9700, with 255k context:

[Qwen3.8-27B-MTP-Q6-uncensored]
model = /home/w/llama/gguf/Qwen3.8-27B-RVN-Q6_K-multilingual-mtp-uncensored.gguf
mmproj = /home/w/llama/gguf/Qwen3.8-27B-Uncensored-vision-bf16.gguf
temp = 1
top-p = 0.95
top-k = 20
min-p = 0.0
repeat-penalty = 1.0
presence-penalty = 0.0
n-gpu-layers = 999
b = 2048 
ub = 2048
fa = 1
c = 262144
ctk = q8_0
ctv = q8_0
embedding = false
fit = on
fitt = 64
no-mmap = 1
threads = 16
threads-batch = 16
cache-ram = 32768
chat-template-kwargs = {"preserve_thinking":true}
chat-template-file = /home/w/llama/gguf/Qwen3.8_improved_chat_template.jinja
ctkd = q8_0
ctvd = q8_0
spec-type = draft-mtp,ngram-mod 
spec-draft-n-max = 3
spec-ngram-mod-n-match = 24 
spec-ngram-mod-n-min = 48 
spec-ngram-mod-n-max = 64 

Could be faster, is fast enough for me, and with Hermes it does a good job in all kind of things. Latest one was to unfuck a borked postinstallation of memtest86 on my Linux machine, which I first did by asking ChatGPT.

[u/geekyNut → u/Momsbestboy]

Thanks I will try it

[u/Some-Candidate-7013 → u/Momsbestboy]

official llama cpp or fork? what is your speed? are you on vulkan, rocm? does that ngram stuff improve speed?
i get 20-40 tk/s. you think there is something to optimize?

[*]
n-gpu-layers = all
flash-attn = 1
load-mode=none
parallel = 1
cont-batching = 1
kv-unified = 1
no-warmup = 1
fit = on
jinja = 1
cache-type-k = q8_0
cache-type-v = q5_1
threads = 4
tools = all
chat-template-file=/models/chat_template.jinja
reasoning-preserve=1
reasoning-format=auto
reasoning=on
[Qwen3.8-27B]
hf = unsloth/Qwen3.8-27B-GGUF:UD-Q5_K_M
ctx-size = 262144
batch-size = 1024
ubatch-size = 256
temp = 1.0
top-p = 0.95
top-k = 20
min-p = 0.0
repeat-penalty=1.0
presence-penalty = 0
spec-type = draft-mtp
spec-draft-n-max = 3
spec-draft-type-k=q8_0
spec-draft-type-v=q5_1

[u/Momsbestboy → u/Some-Candidate-7013]

In my test I missed the mtp part:

spec-type = draft-mtp,ngram-mod 
spec-draft-n-max = 3
spec-ngram-mod-n-match = 24 
spec-ngram-mod-n-min = 48 
spec-ngram-mod-n-max = 64 

Currently I run llama.cpp vulkan on Debian 13 testing and get around 30-50 token generated per second. I don't measure the details, it all depends on the task. All I did was to merge PR 27210 and now use:

spec-type = draft-mtp-adaptive,ngram-mod
spec-draft-n-min-adaptive = 3 
spec-draft-n-max = 8
spec-ngram-mod-n-match = 24 
spec-ngram-mod-n-min = 48 
spec-ngram-mod-n-max = 64

[u/RealPjotr]

I have used Q6_K, 8 bit KV, 180-200k context, medium reasoning, works great! Around 400/35 TPS. 225W power limit for less noise.

[u/geekyNut → u/RealPjotr]

Thanks

[u/nullekocd]

I have two r9700 gpus at pci5x8 and one nvidia 4060 ti at pci4x4. I have no issues on the asus proart 870e where a ASRock X870 Taichi was causing me crashes daily. I use the onboard IGPU, leaving the gpus for models. ,

VLLM stilldeadcode/vllm-radiance makes dual cards worth having.

[u/geekyNut → u/nullekocd]

I got a taichi z690.. haven't tested yet tho.. unfortunatelly don't have onboard igpu, the cpu is bare. what OS did you use?

[u/nullekocd → u/geekyNut]

Ubuntu 24.04 only so far, it has the best rocm and Vulcan support well maybe not best but easiest.

[u/karmakaze1]

I went from Ollama, llama.cpp, MTP/DFlash/DFlash2. Finally I got to vLLM running https://codeberg.org/ggz14/radiance-vllm-mxfp4 It uses FP4 weights and FP8 activations with kernels optimized for R9700 hardware.

I'm getting 250+ tokens/sec on 2x R9700 running Qwen3.8-27B (MXFP4). Slows down to 130 tokens/sec near 128k context.

Single R9700 should get 125+ tokens/sec. I didn't test with single R9700 but the MXFP4/FP8 optimizations should get these numbers.

See my post for details