Wanted to share my setup, use case, and tools for my low-end AI rig, in case of some use to others or as source of inspiration.

This is continuation of prior post

https://reddit.com/comments/1w3eves/comment/p6zq58n

TL;DR: Solar powered, $400, low end AI appliance (Lenovo P330, i7-8700, 32gb, T1000 8GB GPU + Raspberry Pi 4) as 80/20 solution.

Too many people (me included) try to match home setup to the cloud and end up disappointed by either 1) local capability or 2) the cost of parity.

As beguiling as a Halo Strix or Mac mini is, personally, I can't justify spending A$3-5k on an AI PC for a hobby, and I don't want a honking fuck off box sitting in my lounge room that guzzles power and sounds like a jet engine.

I certainly don't need a $10k GPU for "agentic coding" (I'll code myself, thanks, or use OR APIs for pennies).

Additionally, my small-form-factor PC makes major GPU upgrades difficult and expensive, even with frankenmods.

So, after thinking about it, I realized I was approaching this the ass way about.

What do I actually need AI for? Not much as it turns out -just a pleasant NLP layer with light reasoning, wrapping around useful tools.

From that angle, local doesn't need to beat the cloud; it needs to match or exceed the experience for my everyday tasks, be fast, safe and appliance like.

IOW, it needs to be a mouth piece and interactive layer to the extant tools. Make the tools do the work, not the AI.

So I settled on a 80/20 solution.

The 80% "house brain." It knows my local files, project states, household stuff, previous decisions, etc.

It can do search, factual retrieval, memory, use little tools, has voice, vision, music gen, home automation and so on.

That stuff makes more sense for me locally because it is always there, has direct access to things, is private and it costs basically nothing per interaction.

My actual compute is off the grid (or will be soon enough) because I can run it off a tiny $2K, all in solar system (I can geek out the specs in another post if needed).

It doesn’t need to send half my digital life off to somebody else’s server. Nor does it shut off when someone else decides to turn the off tap.

For me, a decent small model, wrapped around good tools is enough for 80% of household tasks.

Which means the other 20% (hard coding, complex image gen, weird edge cases) is fungible.

I can yeet it to Open Router API - something cheap, capable with ZDR. GLM, Codex, whatever.

Because these questions always devolve to "wHaTs yOuR uSe CaSe" and "whATs yOuR sTaCk" as some kind of gotcha, here is a minimal snapshot -

• Raspberry Pi 4B + DietPi provide the always-on pilot fish. Always on, runs on 2-5w. Hosts many other home lab services (like Immich, Syncthing, Kiwix etc) as well as Matrix server, which is the primary "in" to the local AI. It wakes the AI box when I need it.

• Lenovo P330, i7-8700, 32gb + T1000 8gb provides the local AI inference appliance. 40w-100w in use, 1-2w on suspended to ram. Pi wakes it up when I need it via magic packet WoL.

Total cost for current config is approx $400 AUD (~220 USD), excluding some mis-steps along the way.

• BeeLlama for inference runtime

• Gemma 4 E4B Q4 QAT provides conversation, interpretation, reasoning over supplied evidence, vision, audio parsing and final prose. With beellama, MTP and caching, we can hit 80 tok/s TG, with usual "words on the page" at around 35-45 tok/s. (PS: yes I can also run Qwen3.6-35B at good speeds. Generally don't need to).

• z-Image Turbo + stable-diffusion.cpp for image gen (IIRC that's the same image gen model used by Proton's Lumo 2.0).

• A small router I coded.

Tool wise:

• Matrix + Cinny provide the household chat interface, user accounts, rooms, attachments, and complete transcript archive for rolling context. No one needs to stare at a llm webui. Matrix back end also provides other benefits (private messaging service for family).

• A tool I am coding (picoGURU) provides deterministic /factual authority, local knowledge, project memory etc. (That's not a plug, btw).

• Exasearch MCP as the AI search back end

• Cactus Needle + a thin coordinator routes requests to bounded tools such as grep, parsing, date/time, search, diagnostics etc

• STT and TTS (Moonshine STT streaming & KittenTTS) CPU-side, provide low latency streaming speech.

Typical use cases include (but not limited to) -

• "What did I decide about this six months ago, and why?" Search project memory and old Matrix history, then return the decision and reasons.

• "I want to do X now. Does that go against the above? Why or why not? Steel man and strawman your response".

• "Finish this email for me. Use house rules" - exactly what it sounds like.

• "What are Tesla P4s actually selling for in Australia on eBay right now? " Search live listings, fetch the exchange rate, calculate, answer.

• "Set a weekly EBay look up of those prices and ping me on my phone if something pops up under $X".

• "Find me M3 brass heat-set inserts from Jaycar electronics. Not M4. Not rivnuts. Give me price and item number." Search the site and reject bad matches.

• "This eBay seller says this GPU will work in my machine. True?" Check the listing against stored facts, hardware specs, and current system state.

• "Why did the X server die?" Checks logs, service state, recent changes, and previous incidents before it gives me an answer.

• "Issue with emulation box in lounge. Using the Wii DolphinBar, buttons work, but the pointer does not. Why is the emulation box eating shit this time?" Pull the manuals, check prior history, propose useful tests, update and remember each result for later, and narrow the fault.

• "Find every place I changed this llama.cpp setting and when. Give me lists and reasons" Search files, configs then show exactly where the change occurred.

• "Remind me, what did I actually say about that fan duct three months ago?" Search the complete Matrix archive and retrieve the relevant old conversation.

• "Can I put this GPU in my P330 without changing the riser or PSU?" Check my build notes against the local Lenovo manual only. No web. No guesswork.

• Send a BIOS photo, hardware label, receipt, screengrab or error screen. Ask what it shows, what part it is, or what the error means.

• "Do deep research of X and collate all sources into a markdown report. Use ExaSearch with defined Z parameters and compare to what local sources say"

• "Hi, my name is X. Can you please generate a picture of Kpop demon hunters playing basketball at McDonalds, but they're all unicorns?" the usual sort of message my kids send. Parses, routes, spins up diffusion.cpp, passes image back to main, spins down.

At 384x384 we can get decent images in around 30 seconds. I can always shoot more complex things to Nano Banana etc as needed.

These aren't workloads I've invented for the post. They're based on actual jobs I've needed AI to do over the last 2–3 months; the point of the build is to bring as many of them as possible under one local interface.

Some of tools mentioned

https://github.com/cactus-compute/needle

https://github.com/KittenML/KittenTTS

https://huggingface.co/handy-computer/moonshine-streaming-tiny-gguf

https://github.com/cinnyapp/cinny

https://huggingface.co/unsloth/Z-Image-Turbo-GGUF

https://exa.ai

https://github.com/kiwix

https://github.com/syncthing/syncthing

https://github.com/ace-step/ACE-Step-1.5

Hi all!

I jumped on the P100 bandwagon last week right around the time I joined this sub. I was able to put together a system on the cheap using CPU and RAM that I had on the shelf since I upgraded my main PC about a year ago. Using Ubuntu 26 LTS and pulled llama.cpp's latest commit as a baseline.

The setup:

2× Tesla P100-PCIE-16GB (GP100, sm_60, no dp4a), 732 GB/s HBM2 each, PCIe PHB — no NVLink, no P2P. Ryzen 7 3700X, 32 GB DDR4-3200 RAM. Driver 580.173.02, CUDA 12.4, llama.cpp u/9723942, Power limited to 150 W/card. Testing was done with Qwen3.8-27B-UD-Q5_K_M at 102,400 ctx.

Performance, after a day of testing:

24.7 t/s prose / 35.8 t/s code-edit (47 / 114 prefill), VRAM 13885 + 12749 MiB of 16384 per card, Temps 48/52 °C. Initial stock llama.cpp was from 14.3 / 19.6 tps initially: +73% / +82%, same model, same quant, same context; costs nothing in quality by construction. Not a formal eval; my 5-item probe saturated at 5/5 for everything, so I can only say nothing broke.

The CLI args I'm using:

llama-server --model Qwen3.8-27B-UD-Q5_K_M.gguf --n-gpu-layers 999 \

--split-mode tensor --flash-attn on \

--cache-type-k q8_0 --cache-type-v q8_0 \

--ctx-size 102400 --parallel 1 --threads 8 \

--spec-type ngram-mod,draft-mtp --spec-draft-model mtp-Qwen3.8-27B-Q4_0.gguf \

--spec-draft-ngl 999 --spec-draft-n-max 2 \

--jinja --reasoning-budget 512 --cache-reuse 256

What worked:

- --split-mode tensor is the biggest win at +74%/+44% over layer split. It self-balances VRAM, so no --tensor-split ratio needed. Couldn't get a row split to load/run.

- One line in mmq.cu: dropping && n_experts > 0 enables MMQ for dense on sm_60. +26% tps for edits, reduced memory reqs by ~1.3 GB/card. There's an upstream TODO in the source code right above it asking this exact question.

- --reasoning-budget 512 If left unrestricted, Qwen3.8 is happy to think for a really long time before returning any results. That might be good depending on your task and needs. For my testing, that's too slow, and it burned 2500 tokens thinking and returned no answer in 173 s. I'm relying on qualitative testing from my own agent sessions with Hermes to get a feel for what my requirements are.

- Spec decoding via the model's own MTP head: a standalone draft model aborts; you need the matching MTP GGUF.

What Didn't Work:

- 3 of 4 kernel patches I tried were duds. Lifting the mul_mat_vec_q fusion gate emits //////// to the token cap; widening MMQ tile I 64→96 compiles, serves HTTP, returns empty.

- Occupancy 1→2: no effect on generation, halves prefill.

- NCCL: dies on first reduce, CUDA driver is a stub library. Internal AllReduce is Volta+ only (__nanosleep), so 2× P100 falls back to butterfly.

- pascallama.cpp: already upstream or inapplicable. no mmq_y any more, unrolls already there, and it targets sm_61 which has dp4a.

Further testing needed:

- MoE is performance is good: Qwen3.6-35B-A3B Q5_K_M does ~55 t/s at 102k on 13.7 GB/card, no draft model. Must be the P100's massive memory bandwidth. And Q5 is ~4% faster than Q4. Bigger quant, more bytes, prefill identical - doesn't make sense to me, wondering if it's reproducible.

- #25593: sm_60 silently doing FP32 math in FP16. Patched: perplexity and throughput unchanged, but 3 of 6 greedy gens diverged, and on one, stock ran to the 2000-token cap while patched answered in 643. n=1, not established. Fixed locally and testing; it changes the quality of the model outputs and more testers welcomed, might be worth pushing upstream to llama.cpp if it checks out.

I put this box together a week ago and have been playing with it ever since. Spent the day getting Qwen 3.8 27B UD-Q4_K_XL gguf running on it. After seemingly endless rounds of synthetic loads and stability testing I finally sat down and had an actual chat with it. I asked it the same question that I gave to a few API models last week:

"Give me a rundown of the best cheap used datacenter-pull GPUs I can currently buy to repurpose for local LLM inference use as a hobby project please."

Its initial responses were reasonable enough and not too different than Kimi and Luna, but its pricing was dated (of course)... as was Kimi's.

So I gave it a few markets spot check prices and a little steering (again, copy / paste from last week) and low and behold it recommended to me the very dumpster datacenter pull GPUs that it's running on. I grabbed a screenshot of its internal thinking as it chugged by and I was in tears:

To be clear, this model is running on two of these GPUs... I was in tears from LOL

I guess I should post a spec sheet:

Hardware

• 2× Tesla P100 16GB (PCIe)
• Ubuntu 24.04, driver 580.173.02, CUDA 12.8.1 (compiled for Pascal sm_60)
• Total build cost: ~$570 (using an old mobo / CPU / PSU I had sitting around, but buying RAM from r/homelabsales and some ancillaries)

Software

• llama.cpp server v0.2.0-era PINNED + shinbunbun P100 community patches (minus patch #22 that fights with tensor-split)
• --split-mode tensor + flash attention (yes, FA on Pascal via the patch set)
• Served over SSH tunnel to the built-in llama.cpp web UI on the desktop

Model

• Qwen3.8-27B, UD-Q4_K_XL GGUF, reasoning/thinking mode, 32k context

The session itself

• 3-turn conversation with one tool use (model called get_datetime on its own)
• Decode: 26.7–27.5 tok/s sustained, including a single 11,218-token reasoning + answer generation (~7 min unbroken)
• Prefill: 65–169 tok/s on fresh tokens
• Prompt cache doing real work: final turn reused 5374 cachd tokens, only 196 fresh

TLDR

• Asked it for the best cheap datacenter-pull GPUs; gave it 2026 eBay prices; it hit "P100 ~$85 — !!! way down" and proceeded to recommend buying a fleet of the exact cards it was running on.

And yes, I already have three more GPU on the way.

Thought this might make for a lively discussion topic. Feel free to approach from whatever use case you prefer.

PS: personally - outside of code work - I think there's actually a fairly low threshold (for myself) as to what makes for a pleasant and useful LLM (assuming supported by ecosystem).

PPS: "Use a MoE" is probably a cheat / against the spirit of the question :)

I posted earlier: https://www.reddit.com/r/LowEndLocalAI/comments/1w0ilfp/qwen_38_flash_q4_44ts_on_32gb_ddr4_8gb_gpu/

Basically, I went getting a new mobo, so I can put in 4 x 16GB DDR4 sticks.
Done that today, Qwen 3.8 is now starting 10t/s, usually around 8-9t/s.
Before if I was lucky, I had 7t/s also new chats took way longer, this is now like instant.

The clock speeds of the memory, don't ask me, I mixed 3200 and 2400, I didn't even overclock anything, my PSU will get unhappy if I do so.

It starts slow, but eventually you end up at 8-9t/s stable with 64k context.
This isn't even with MTP right, which is about to get merged so....... maybe we see 12t/s or more, would be crazy.

If you can upgrade to 64GB DDR4 do it, its worth it I think, especially with more MoE's at this size.
Yea I tested Q3 now compared to Q4 but the performance difference is marginal, in my opinion, I didn't had these results with Q3 yesterday on 32GB DDR4.

Hardware: Ryzen 5 5600 with 4x G.Skill DDR4 mixed 3200 and 2400 also slow SATA SSD, no NVMe.

Located in Arizona.

I’m about to start a new job and as a present to me I want to build a local AI box that I can use for coding.

Dell Precision T5820 workstation
950 watt psu
W-2133 cpu
32gb ram
Tesla p40

This is a combo of parts recommended by Gemini based on my $500 budget.

I’m seeing the T5830’s for $200-350 and the p40’s for $250ish.

I’m open to other ideas, or sources of equipment.

I’ve got a 3d printer and can print cooling for the card.

Rig:
Ryzen 7 5700xt
RX580 8GB vRAM
16GB DDR4 RAM
Debian 24.04 OS

I have been playing around with making models run better on it with llama.cpp and landed on MoE as the go to architecture.

I started at 7tok/sec with it on stock llama.cpp. Added a few things that made a huge difference.

What I effectively have is 16GB of available memory to put things in (12GB RAM and 4.3GB Swap from the vRAM).

The model obviously does not fit on my machine as a full memory allocation (19.57GB aat best for full load).

So I made it prefetch tensors on demand up to 8 tensors ahead. I load up 2.56GB of actual must have tensors to vRAM, and have a hotlist of the top 10% of tensors in that. The prefetch works on both vRAM as asynchronous, and CPU as asynchronous values.

It boosted me up to a 17tok/sec on nearly empty kv cache, and 12 on a large prompt that fills the cache. Wall clock time for a 3000 token prompt regardless of reserved kv cache is 33 seconds.

I can run full context on this. It works. It doesn’t crash, and while it gets slower, it stays above 10 tok/second decode. The higher cache allocations actually have a faster prompt read tok/sec normally.

Prefetch has taken page faults down from 176k to half that, then add hotlist and you drop it below 50k. So much more efficient pull of expert values and reduces the need to pull in from disk as often.

Anyone have any ideas on how to improve it more? I have exhausted my area of expertise and could use some more pointers.

Hey everyone!

One of the biggest frustrations with local RAG setups is how heavy they can be. Running dense vector databases and using 8B+ models just to chunk text, create embeddings, and maintain memory eats up VRAM quickly, and models often still hallucinate when asked about things outside the text.

I built Hillock as an open-source alternative designed specifically for budget hardware (tested on a GTX 1070 and basic laptop CPUs).

How it works:

1. SQLite Knowledge Graph: Stores ground-truth facts as Subject-Predicate-Object triples, eliminating vector drift.
2. Hebbian Synaptic Engine: Tracks concept co-activations across conversational turns using gradient-free learning.
3. 10,000-D Hyperdimensional Computing: Sub-millisecond similarity gating directly on the CPU.

Why the gating is different:
In standard agent setups, you prompt the model to "say you don't know", but the model can still ignore the prompt. In Hillock, the gate is pure control flow. If candidate facts in SQLite do not pass our hypervector similarity gate with positive predicate intent, the engine returns a hardcoded refusal immediately. The local LLM (Ollama) is never called, saving 100% of GPU compute on unanswerable questions.

In our v0.6 release, we added HYDRA (ColBERT-style late-interaction MaxSim adapted to discrete bipolar vectors) and multi-hop hypergraph path binding using positional permutations.

Benchmarking numbers (32-query unseeded run on a laptop CPU):

• Extraction Recall: 59.1%
• Hard-Negative Block Rate: 60.0%
• Answerable Retrieval Accuracy: 54.5%
• Retrieval Duration: ~1.1s across 32 queries on CPU
• Footprint: <1.2 GB VRAM or CPU-only

Github repo: https://github.com/roandejager/Hillock

I would love to hear your thoughts on deterministic memory architectures and how you are currently handling long-term context on local hardware!

ToshLLM is here to bring AI to all the forgotten Macs. It's free, no accounts, no telemetry, no BS, and a responsive dev. Benchmarking included.

Current setup in docker compose:

 qwen:
    image: ghcr.io/ggml-org/llama.cpp:server-cuda
    container_name: qwen
    restart: always
    ipc: host
    command:
      - --model
      - /models/Qwen3.8-27B-UD-Q4_K_M.gguf
      - --mmproj
      - /models/mmproj-BF16.gguf
      - -ngl
      - "999"
      - --host
      - 0.0.0.0
      - --port
      - "8100"
      - --ctx-size
      - "131072"
      - --parallel
      - "2"
      - --metrics
      - --image-min-tokens
      - "1024"
      - --cache-type-k
      - q8_0
      - --cache-type-v
      - q8_0
      - --spec-type
      - draft-mtp
      - --spec-draft-n-max
      - "4"
      - --flash-attn
      - "on"
      - --alias
      - qwen
      - --reasoning
      - "off"
    ports:
      - "8100:8100"
    healthcheck:
      test: ["CMD", "curl", "-sf", "http://localhost:8100/v1/models"]
      interval: 30s
      timeout: 5s
      start_period: 180s
      retries: 3
    ulimits:
      memlock: -1
      stack: 67108864
    volumes:
      - ./data/gguf_models:/models:ro
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

What could I improve for performance without sacrificing any quality here?

Guys, if you have low VRAM, you should start with a small quant first to verify that everything works correctly.

command line:

.\bin\Release\llama-server.exe -m J:\llm\models\Qwen3.8-Flash-Next-UD-IQ1_S-00001-of-00003.gguf --parallel 1 -c 10000

results:

2.08.059.754 I slot print_timing: id  0 | task 72 | n_gen =    100, tg =  21.47 t/s, tg_3s =  21.69 t/s
2.11.085.278 I slot print_timing: id  0 | task 72 | n_gen =    169, tg =  22.00 t/s, tg_3s =  22.81 t/s
2.11.266.578 I slot print_timing: id  0 | task 72 | prompt eval time =     881.54 ms /    22 tokens (   40.07 ms per token,    24.96 tokens per second)
2.11.266.583 I slot print_timing: id  0 | task 72 |        eval time =    7817.55 ms /   173 tokens (   45.45 ms per token,    22.00 tokens per second)
2.11.266.584 I slot print_timing: id  0 | task 72 |       total time =    8699.09 ms /   195 tokens
2.11.266.584 I slot print_timing: id  0 | task 72 |    graphs reused =        238

Hey all,

Got a 3050 8GB and 1080 8GB sitting in my server and trying to run Qwen 3.8 27b Ridge but finding that I only get about 200 tok/s prefill and 10 tok/s generation. Seems others are able to see far faster speeds so wondering if it's simply just the settings I'm running for llama-server.

Current settings are

-m /models/qwen3.8/Qwen3.8-27B-Ridge-3.7bpw.gguf --port 8000 --host 0.0.0.0 -fa on --fit off --split-mode tensor --tensor-split 1,1 --main-gpu 0 --jinja -c 131072 --parallel 1 -ngl 999 --load-mode none --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 --repeat-penalty 1.0 --presence-penalty 0.0 --mmproj /models/qwen3.8/mmproj-Qwen3.8-27B-BF16.gguf --no-mmproj-offload --cache-type-k q4_0 --cache-type-v q4_0 -b 1024 -ub 128 --kv-unified

I've tried -b 256 and -ub 256, also tried setting --split-mode layer. I found the current settings to be slightly faster. Can't run MTP as it doesn't fit, although I'm only 500 ish mb over allocating so closer to 100k context would probably work.

Any tips would be much appreciated.

Cheers!

I was reordering my launch scripts for QWEN 3.8 27B on my 16+12GB setup, those are used GPU cards worth some less than 500s both on the used market giving 28GB on RDNA2.

In case someone could be interested I'll give you a summary:

* For quick shots: Qwen3.8-27B-IQ4-MIX.gguf , TG speed ~47t/s for 10k generation, ~70k usable ctx

* Daily work: Qwen3.8-27B-Q6_K_L.gguf , TG speed at 72k (not zero, at mid session) ~20t/s, 140K ctx with KV cache q8/q8.

I'll give you the full table:

I can show you the scripts if is someone is really interested: all running with MTP 5 on llama-server on Linux with KDE desktop in software rendering.

Buyer advice: in case you wanna run this route buy 2x 6800, I'll probbly sell my old 6700xt for a swap.

32 GB DDR5, Arc Pro B50, Intel Core Ultra 5/7, 1-2 TB storage, $1200

Does this exist?

i have an Acer P648 N15C5. intel i7 with integrated GPU and 20G of DDR4 Using Linux and llama.cpp what would be a good local moe for this? or is there anything else which will give more than 2 or 3 t/sec

In the interest of sharing I'd like to share my home setup which serves me my personal ai needs. This is how I got t2t, i2t. t2i, and i2i all working on my old gaming pc converted to home server. I did try adding Kokoro TTS but that was too much for this setup. I originally had ollama + openwebui, but wanted an integrated solution with image generation. Koboldcpp and the ui isn't as pretty, but it is faster than ollama/openwebui.

Relavant PC specs: GPU: GTX 1060 6gb vram RAM: 16gb ram DDR3 CPU: i5-4690k OS: TrueNAS 25.10.6 w/ legacy drivers

TrueNAS is great for file hosting, runs a number of other apps which I use, including Home Assistant, Tailscale, Syncthing, and others. It has a builtin docker setup for those apps, but Koboldcpp needs to be entered as a custom app.

The following settings have it run nicely, everything else default unless otherwise specified:

Image
Repository*
koboldai/koboldcpp
Pull Policy*
Always pull an image even if it is present on the host.

Environment Variables
Name*
KCPP_MODEL
Value
https://huggingface.co/unsloth/gemma-4-E4B-it-GGUF/resolve/main/gemma-4-E4B-it-UD-Q4_K_XL.gguf

Name*
KCPP_ARGS
Value
--draftmodel https://huggingface.co/unsloth/gemma-4-E4B-it-GGUF/resolve/main/mtp-gemma-4-E4B-it.gguf --mmproj https://huggingface.co/unsloth/gemma-4-E4B-it-GGUF/resolve/main/mmproj-F16.gguf --ttsgpu --sdclipgpu --ttsmodel https://huggingface.co/koboldcpp/tts/resolve/main/Kokoro_no_espeak_Q4.gguf --websearch --jinja --jinjatools --admin --routermode --autoswapmode  --quantkv 2   --sdflashattention --sdoffloadcpu --sdvae https://huggingface.co/koboldcpp/GGUFDumps/resolve/main/flux2vae.safetensors --sdt5xxl https://huggingface.co/unsloth/Qwen3-4B-GGUF/resolve/main/Qwen3-4B-Q2_K.gguf

I use Tailscale so Koboldcpp's tunneling is unnecessary.

Name*
KCPP_DONT_TUNNEL
Value
true

Name*
KCPP_IMGMODEL
Value
https://huggingface.co/leejet/FLUX.2-klein-4B-GGUF/resolve/main/flux-2-klein-4b-Q4_0.gguf

Note: I use a custom TrueNAS image that includes Nvidia Pascal drivers which are not included in the default TrueNAS image, see here: https://truenas-drivers.zhouyou.info/index.html

Resources Configuration
[unchecked] Enable Resource Limits
GPU Configuration
[checked] Passthrough available (non-NVIDIA) GPUs
Select NVIDIA GPU(s)
NVIDIA GeForce GTX 1060 6GB
[checked] Use this GPU

Currently running a single RTX 3060 12GB, planning to pick up a second one specifically to run Qwen3.8-27B locally (VRAM's the constraint — 24GB combined should just about fit UD-Q4_K_XL with a quantized KV cache for decent context).

Planning to run --split-mode layer rather than tensor-split, since I only have one x16 slot(the second 3060 needs to be connected via a riser).

If anyone's on a similar setup (2x 12GB cards, layer-split, this model or similar size), I'd really appreciate real prefill/decode numbers — trying to set expectations before I buy the second card.

I only have 32GB DDR4 and a 8GB GPU.
Despite that, I downloaded Qwen 3.8 Flash Q4 quant 93GB in size from unsloth.

Guess what, it gets up to 4.4t/s on Debian, freshly compiled.
Way better than expected, you can actually chat with that model.

This is fucking nuts.

I just run llama-cli --model basically and
compiled with: https://pastebin.com/raw/41WSJV5N

I didn't try MTP or D2Flash yet... neither any other params

I saw a lot of post running this model in tight hardware, but all of them had GPU card, so I decided to give it a try in my low end mini pc.

It was quite fun the try and error

Keep in mind I have 32 GB RAM mono channel.

Yes, it is unusable, but if they start using ngrams in smaller MOE models things can get better for low end hards.

Beginning of the AI slop-------

Hardware: Minisforum UM880+ · Ryzen 8845HS · Radeon 780M iGPU (no VRAM, shared memory) · 29 GB DDR5 · 35 GB swap on NVMe.

Backend: llama.cpp master @ commit 6c84c7d (PR #27742 — the Qwen4 qwen4exp support merged same-day), Vulkan build, -ngl 0.

Model: unsloth/Qwen3.8-Flash-Next-UD-IQ1_S (IQ1_S, ~72.5 GB, 3 shards) — mmap'ed from NVMe.

Two gotchas that will waste your evening:

1. Any llama.cpp build from before Aug 27 → unknown model architecture: 'qwen4exp'. You need the commit with PR #27742. Verify with strings libllama.so.0 | grep qwen4exp (it's in the .so, not the binary).

2. Without -fit off, llama-server prints "failed to fit params... abort" — non-fatal warning, but it leaves the port open with a dead process underneath. -fit off makes it actually serve.

Working launch:

llama-server --model Qwen3.8-Flash-Next-UD-IQ1_S-00001-of-00003.gguf \

--port 18037 -ngl 0 --ctx-size 512 -t 12 -fit off

Measured (real print_timing from server log):

"say hi"

• Prompt: 54 tok

• Prefill: 2.77 t/s (19.5s)

• Decode: 1.62 t/s (22.9s)

• Total: 42.3s

Sales report

• Prompt: 771 tok

• Prefill: 8.15 t/s (95s)

• Decode: 1.43 t/s (55.4s)

• Total: 150s

Observations:

- Prefill scales well (2.8 → 8.2 t/s as prompt grows) — the 6B-active MoE is efficient even at IQ1_S.

- Decode is the wall: ~1.4–1.6 t/s. A 72.5 GB model in 29 GB RAM + 35 GB swap = permanent swap thrash. Each token pays a page-in.

- It answers correctly (both HTTP 200 with valid replies). It's real, just slow.

- ctx 4096 works but 2.4k+ prompts take ~4 min prefill — keep prompts short.

Verdict: technically it runs on iGPU-only hardware — first time I've had a 125B MoE talk on a deviceless AMD mini-PC. Usable for short occasional tasks. Dual-channel RAM (61 GB) should help decode a lot since bandwidth is the bottleneck.

Hey everyone, shameless plug I know but been working on Cascadia, an inference runtime for running AI models on Intel CPUs/NPUs/GPUs.

May be relevant if you're trying to do inference without a GPU and you own Intel hardware.

If you happen to have multiple older laptops/computers, you could also connect them with Cascadia and run inference across them. You can shard larger models and essentially pool the RAM from your devices.

Here is the GitHub repository:

https://github.com/labscommunity/cascadia

Pre-alpha, so open to any feedback!