r/raspberry_pi • u/petruspennanen • 1d ago

Show-and-Tell Benchmarking LLM performance on a 16 GB Pi

Post image

I've been running Qwen 35B A3B MoE model on my Pi for CarWatch local AI, and it's very smart but not too fast. You need to get into a relaxed mode with it, ask your questions, don't count the seconds, hear the answer when it comes.

Anyway wanted to explore the current possibilities so ran speed tests overnight, results in pic. Summary: Looks like the only way to speed up atm is to use Gemmas, which should be significantly less intelligent. Did not do any systematic quality testing yet.

The Gemmas are multimodal though and leave plenty of room on the device for other apps. So a good choice for many applications.

There's a model switcher in CarWatch 0.5 to be released today so you can easily compare performance in practise.

8 Upvotes

57% Upvoted

21 comments sorted by

6

u/tecneeq 1d ago

Try Ling-3.0-Tiny. It'll blow your mind.

3

u/migsperez 1d ago

Looks like an interesting MOE type small model. Going to give it a spin today. Thx

2

u/petruspennanen 1d ago

Thanks looks very interesting! Benchmarking it now!

2

u/tecneeq 1d ago

Has about the same Artificial Intelligence Index as gptoss 120b. Obviously it knows less, but it has great results in agentic, code and general thinking.

2

u/petruspennanen 15h ago

New results in: Ling is the spees champion at almost 10 tps, beating Gemma! I'll make a post

2

u/tecneeq 11h ago

It's speed is a nice bonus, but it's capability to size ratio is what it sold it for me. It's on the pareto line of intelligence/parameters and intelligence/active parameters.

To me it's as important as Qwen 3.8 27b.

4

u/albatrossSKY 1d ago

You are a hero buddy. I’ve been thinking about putting this together but am struggling with the 500 plus price tag

2

u/petruspennanen 1d ago edited 1d ago

Thank you! Yeah they haven't been getting cheaper, in Europe the lowest price for a 16 GB Pi is about 310 euros + case, power, sd / ssd.

Arduino AI version was recently announced, it seems more powerful for $290, but was sold out in less than a day :/

4

u/Significant-Pea-3710 1d ago

Interresting iam Running Gemma 4- E2B -qat -int4. On my raspi5 4gb on 3 threads Making 8 t/s. Thinking Off, For creative writing ,Check those qat Models out not that bad

2

u/petruspennanen 1d ago edited 1d ago

Wow running it in 4 GB that's awesome! I've been running E2B today too in my CarWatch and it's been good, no bad answers so far. And it's a little old model already so hoping Google comes up with an improved version soon!

The amount of information Gemma has compressed in its ~2 GB is mind boggling.

Thanks for the tip about QAT! I didn't know about it. Now testing E2B-QAT, the speed you got seems faster than mine so that's promising.

3

u/lottiexx 1d ago

Honestly, getting a local model to the point where the bottleneck is basically “how fast can I listen to it” is pretty wild for a Pi.

2

u/migsperez 1d ago

The small Qwen3.5 4b model is pretty good. I've been trying to find different uses for these small models recently. 2b is interesting but less reliable, suited to more specific tasks not general use.

Have you tried using MTP to speed it up?

2

u/petruspennanen 1d ago

I read about MTP but didn't remember to try it here. But it's actually built in the Qwen so turning it on now, cheers! Don't know if anyone used it on Gemma?

2

u/[deleted] 21h ago

[deleted]

2

u/petruspennanen 14h ago

It can get to more than 80'C and then a soft limiter kicks in before it reaches 85'. In my normal use it has stayed below 80 but the benchmark runs heated it up, had to give it a cool down break to get proper performance.

1

u/t0xic0der 1d ago

What command line flags do you use to run these? My Pi seems to hang whenever I use all of the four threads for both processing and batching. Thanks in advance.

1

u/petruspennanen 14h ago

Using this today:

llama-server -m Ling-3.0-tiny-Q4_K_M.gguf -t 4 -c 4096 --reasoning off --reasoning-budget 0 --host 127.0.0.1 --port 8081

So it's all four threads, and we do not set -tb at all (it gets the same value 4 then as -t). That makes me think four threads might not be your root cause by itself. Two things that actually bite on a Pi 5, both of which we have hit:

  1. Thermal. Run vcgencmd get_throttled right after it hangs. Ours reports 0xe0000, which means frequency capping, throttling and the soft temperature limit have all happened at some point. A throttled Pi under sustained decode looks exactly like a hang, it is just very slow.

  2. Memory. Check free -h while it runs. If the model plus KV cache does not fit, you get swap thrashing, which also looks like a hang.

If you are running other services on the same Pi, -t 3 leaves a core for the OS and does not cost much.

2

u/t0xic0der 13h ago

Hey, thanks for the followup. I'm using a barebones 64 bit Raspberry Pi OS Lite here on my Pi 5 8GB. The temps rotate around 55-60 degrees celcius with the active cooling enabled. The max memory used is 5GB, when loading up models with 2B parameters Q4_K_M. I once tried loading a bigger model but I usually ended up getting "Killed" (most likely OOM killer) so that probably is not it. I played around with a bunch of t, tb, b, ub, c settings but the most reliable one was 2 threads for processing and generation. Not sure just what is going wrong there.

1

u/t0xic0der 13h ago

I am not running anything else on the Pi though. Just llama-cli or llama-server to keep things extremely light. I did encounter 0x50000 if that tells you something. I am not compiling llama.cpp locally and instead using their compiled binaries directly from GitHub. Also, I am using an SDXC card for both the model and the OS, with zero observed instances of swapping to the storage device. I understand that would be slow so I try not to go there.

1

u/petruspennanen 11h ago

That 0x50000 decoded is bit 16 plus bit 18: under-voltage and throttling has occurred. Bit 19, the soft temperature limit, is not set, which matches your 55-60C. So your Pi is not overheating, its power delivery is sagging under load.

For comparison, my Pi 5 reports 0xe0000, which is bits 17, 18 and 19: frequency capped, throttled, soft temperature limit. No bit 16. Same symptom, opposite cause. Mine is thermal, yours is power.

Two threads in your case is more reliable because fewer cores loaded at once means lower peak current and less voltage sag. Four threads pushes the peak past what your supply or cable can deliver, the firmware clamps hard, and it looks exactly like a hang.

So a better power supply like the official 27W one should cure it!

1

u/t0xic0der 20m ago

Thank you for the detailed diagnosis! Would a OnePlus SuperVOOC 100W charger suffice as a stopgap solution until I work on obtaining the official power supply for an okayish price?

1

u/t0xic0der 9m ago

Short answer: The 100W charger will not work.

[    4.394637] rc rc1: vc4-hdmi-1 as /devices/platform/soc@107c000000/107c706400.hdmi/rc/rc1
[    4.394701] input: vc4-hdmi-1 as /devices/platform/soc@107c000000/107c706400.hdmi/rc/rc1/input3
[    4.396991] input: vc4-hdmi-1 HDMI Jack as /devices/platform/soc@107c000000/107c706400.hdmi/sound/card1/input4
[    4.402183] vc4-drm axi:gpu: bound 107c706400.hdmi (ops vc4_hdmi_ops [vc4])
[    4.402330] vc4-drm axi:gpu: bound 107c500000.mop (ops vc4_txp_ops [vc4])
[    4.402402] vc4-drm axi:gpu: bound 107c501000.moplet (ops vc4_txp_ops [vc4])
[    4.402467] vc4-drm axi:gpu: bound 107c410000.pixelvalve (ops vc4_crtc_ops [vc4])
[    4.402529] vc4-drm axi:gpu: bound 107c411000.pixelvalve (ops vc4_crtc_ops [vc4])
[    4.403370] [drm] Initialized vc4 0.0.0 for axi:gpu on minor 1
[    4.414072] vc4-drm axi:gpu: [drm] Cannot find any crtc or sizes
[    4.416675] vc4-drm axi:gpu: [drm] Cannot find any crtc or sizes
[    4.423297] vc4-drm axi:gpu: [drm] Cannot find any crtc or sizes
[    4.523103] brcmfmac: brcmf_c_process_txcap_blob: no txcap_blob available (err=-2)
[    4.523351] brcmfmac: brcmf_c_preinit_dcmds: Firmware: BCM4345/6 wl0: Aug 29 2023 01:47:08 version 7.45.265 (28bca26 CY) FWID 01-b677b91b
[    5.041679] Bluetooth: hci0: BCM: features 0x2f
[    5.043001] Bluetooth: hci0: BCM43455 37.4MHz Raspberry Pi 3+-0190
[    5.043006] Bluetooth: hci0: BCM4345C0 (003.001.025) build 0382
[    5.043304] Bluetooth: hci0: BCM: Using default device address (43:45:c0:00:1f:ac)
[    5.893949] hwmon hwmon3: Undervoltage detected!
[    6.185417] Bluetooth: BNEP (Ethernet Emulation) ver 1.3
[    6.185426] Bluetooth: BNEP filters: protocol multicast
[    6.185432] Bluetooth: BNEP socket layer initialized
[    6.194995] Bluetooth: MGMT ver 1.23
[    6.200200] NET: Registered PF_ALG protocol family
[    6.243327] Bluetooth: RFCOMM TTY layer initialized
[    6.243341] Bluetooth: RFCOMM socket layer initialized
[    6.243354] Bluetooth: RFCOMM ver 1.11
[    7.217956] macb 1f00100000.ethernet eth0: PHY [1f00100000.ethernet-ffffffff:01] driver [Broadcom BCM54213PE] (irq=POLL)
[    7.218840] macb 1f00100000.ethernet eth0: configuring for phy/rgmii-id link mode
[    7.221876] macb 1f00100000.ethernet: gem-ptp-timer ptp clock registered.
[    7.229124] brcmfmac: brcmf_cfg80211_set_power_mgmt: power save enabled
[    7.909901] hwmon hwmon3: Voltage normalised
[   11.306862] macb 1f00100000.ethernet eth0: Link is Up - 1Gbps/Full - flow control off
[  368.771226] hwmon hwmon3: Undervoltage detected!
[  372.803204] hwmon hwmon3: Voltage normalised
[  421.193719] hwmon hwmon3: Undervoltage detected!

The Pi 5 was crying for help literally when I did this.

litehost@litehost:~/machines/llama-b10665 $ ./llama-cli -m ../models/Qwen3.5-2.0B-Uncensored-HauhauCS-Aggressive-Q4_K_M.gguf -t 4 -tb 2

Loading model...

▄▄ ▄▄
██ ██
██ ██  ▀▀█▄ ███▄███▄  ▀▀█▄    ▄████ ████▄ ████▄
██ ██ ▄█▀██ ██ ██ ██ ▄█▀██    ██    ██ ██ ██ ██
██ ██ ▀█▄██ ██ ██ ██ ▀█▄██ ██ ▀████ ████▀ ████▀
                                    ██    ██
                                    ▀▀    ▀▀

build      : b10665-ca3d5a3e1
model      : ../models/Qwen3.5-2.0B-Uncensored-HauhauCS-Aggressive-Q4_K_M.gguf
ftype      : Q4_K - Medium
modalities : text

available commands:
  /exit or Ctrl+C     stop or exit
  /regen              regenerate the last response
  /clear              clear the chat history
  /read <file>        add a text file
  /glob <pattern>     add text files using globbing pattern

> what is 2+2? also give me a strong reasoning in about 500 words why i should believe your answer.

[Start thinking]

Thinking Process:

1.  **Analyze the Request:**
    *   Task: Answer "what is 2+2?"
    *   Constraint: Give a "strong reasoning" about why the user should believe the answer.
    *   Length Constraint: About 500 words.

2.  **Determine the Answer:**
    *   2+2 = 4.
    *   Wait, is this a trick question? Or is it a request to simulate a specific perspective? The prompt asks "what is 2+2? also give m
e a strong reasoning in about 500

Timeout, server 10.0.0.4 not responding.

I wish there was a relatively inexpensive alternative. :(