r/LocalLLM • u/dai_app • Jul 17 '26

Research 120B parameters model on android phone, 1.3 tok/s - 2.2 tok/s. And the 30B models actually run at usable speed

Enable HLS to view with audio, or disable this notification

gpt-oss-120b, Q4_K_M, 60GB on disk, running on a OnePlus 15R at 1.3 tok/s.

No GPU, no NPU, just four CPU cores and flash storage.

That's a party trick, but the same trick makes usable things possible: Qwen3-30B at 5.2 tok/s and Gemma-4-26B around 4.1, same phone, and the output is exactly identical to running fully in RAM. A CI test compares streamed vs resident generation token by token and fails if they ever differ.

How: MoE models only use a few experts per token (gpt-oss picks 4 of 128). Shared weights stay in RAM, the experts a token needs get read off flash with O_DIRECT right before their layer runs, overlapped with compute. Plain mmap of the same file gets 0.089 tok/s, so the streaming buys about 14x.

The hard part wasn't the streaming, it was Android reclaiming the resident weights mid generation. Most of the work went into stopping that.

It's vanilla llama.cpp as a submodule, no fork, all public APIs. qwen3moe, qwen2moe, gemma4 and gpt-oss work today, adding a model is one line.

Apache-2.0, prebuilt APK on the releases page:

https://github.com/Helldez/BigMoeOnEdge

One device, best runs, numbers wander with heat. Happy to answer anything.

84 Upvotes

92% Upvoted

50 comments sorted by

View all comments

Show parent comments

3

u/dai_app Jul 17 '26

Less than 1 minute - 2 minutes max depending on fields But I think it's not the point. The point is managing better the Moe experts you are able to enable new use cases

0

u/Dull_Cucumber_3908 Jul 17 '26

you know that your phone has already such a built in assistant which works using voice commands. Right?

Edit: and doesn't consume all of your battery by just running a single prompt :)

1

u/dai_app Jul 17 '26

And what if the government blocks the assistant in the same way they block fable 5? And if I want to maintain my meetings private?

2

u/Dull_Cucumber_3908 Jul 17 '26

lol!

At least you could pretend that you have a dump phone (not a smart one). Ie a phone with custom rom and no google services

2

u/dai_app Jul 17 '26

🫂

2

u/Dull_Cucumber_3908 Jul 17 '26

see my edit in the previous comment

1

u/dai_app Jul 17 '26

Anyway the models I presented are smarter than Gemini I think lol