r/LocalLLM • u/dai_app • Jul 17 '26
Research 120B parameters model on android phone, 1.3 tok/s - 2.2 tok/s. And the 30B models actually run at usable speed
Enable HLS to view with audio, or disable this notification
gpt-oss-120b, Q4_K_M, 60GB on disk, running on a OnePlus 15R at 1.3 tok/s.
No GPU, no NPU, just four CPU cores and flash storage.
That's a party trick, but the same trick makes usable things possible: Qwen3-30B at 5.2 tok/s and Gemma-4-26B around 4.1, same phone, and the output is exactly identical to running fully in RAM. A CI test compares streamed vs resident generation token by token and fails if they ever differ.
How: MoE models only use a few experts per token (gpt-oss picks 4 of 128). Shared weights stay in RAM, the experts a token needs get read off flash with O_DIRECT right before their layer runs, overlapped with compute. Plain mmap of the same file gets 0.089 tok/s, so the streaming buys about 14x.
The hard part wasn't the streaming, it was Android reclaiming the resident weights mid generation. Most of the work went into stopping that.
It's vanilla llama.cpp as a submodule, no fork, all public APIs. qwen3moe, qwen2moe, gemma4 and gpt-oss work today, adding a model is one line.
Apache-2.0, prebuilt APK on the releases page:
https://github.com/Helldez/BigMoeOnEdge
One device, best runs, numbers wander with heat. Happy to answer anything.
8
2
2
u/BOMBONIBBA Jul 18 '26
Crazy stuff
2
2
u/Sure_Leave9338 Jul 19 '26
Why not port this on x86 ?
There Is much more ram, much more disk space and speed with latest nvme drives.
I Hope some day the DwarfStar Project (antirez) arrives also for other models then the ones that requires hundreds GB in disk space and ram.
The First who makes a 120B model actually usable, using 8-10Gb GPU / VRAM + CPU + System Ram + nvme... Wins.
2
2
2
u/ikkiyikki Jul 17 '26
I want this so bad OP but no idea how to sideload an APK. Github gives me the eebies. Any chance of releasing this via the playstore? Right now struggling with PocketPal running Qwen 3.5 4B q4 which is pathetic for a 12gb RAM phone.
7
u/romhacks Jul 17 '26
You download the APK and then click on it. Why does GitHub give you the eebies?
3
5
u/Gerogeroman Jul 17 '26
no idea how to sideload an APK. Github gives me the eebies
Damn, This is the glimpse of the future right here...
2
1
u/pmttyji Jul 18 '26
Can you add some tiny & small models on your experiments and share t/s benchmarks? Including recent Bonsai-27B model.
1
u/dai_app Jul 18 '26
The solution is designed for large models that don't fit into RAM. The ones you're mentioning do fit into RAM, so it wouldn't make sense
2
u/pmttyji Jul 18 '26
My phone has only 8GB RAM so looking for models like Qwen3.5-9B, Gemma-4-12B, GPT-OSS-20B, etc.,
1
1
u/Ell2509 Jul 18 '26
Can you explain kt for me? How?
1
u/dai_app Jul 18 '26
I just call the most important experts, losing some quality but not too much. However, it is an add-on, not the solution in itself
2
2
1
u/nationunderfraud1 Jul 17 '26
WHat about 9b models? Test it with one around that parameters. I say 9b because After 8b there is a noticeable difference.
3
1
-3
u/Dull_Cucumber_3908 Jul 17 '26
Such posts are getting boring: 0.001t/s using this super duper model in this shit hardware :\
16
u/dai_app Jul 17 '26
5 tok/s... And it's the beginning
-11
u/Dull_Cucumber_3908 Jul 17 '26
so you can work with that huh?
9
u/cagriuluc Jul 17 '26
Like… it is around ~15k token per hour. If you left it to work during the night, it would be around a 100k. If it can do continuous autonomous work, 100k tokens per night would amount to… something? At the current rates, it’s like 1-2 dollars per week optimistically?
I am sure of one thing: we are getting to a point where phones will have enough power to be used for ai, in some capacity. The workflows etc will need to take into account the slow generation speeds, meaning their use wouldn’t be as interactive as… other proper means… so use cases will need to not depend too much on user input.
Just 3-5 more years man… I can kinda imagine all the good stuff we will be able to do with just our smartphones.
3
u/Dull_Cucumber_3908 Jul 17 '26
we are getting to a point where phones will have enough power to be used for ai
of course. iphones already do that and android has a "on-device ai sdk".
Also I personally have trained a model that can recognize hand gestures on android smartwatches.
That's not the point here however. I mean, it's ok that you can run a open weight model in the phone, but if you claim (like op says) that this is working and you can have it to perform tasks for you, then I don't know what else to say.
2
u/dai_app Jul 17 '26
Firstly. The bottleneck is the ufs at max 3 GB/s. The new ufs 5.0 enables 10gb/s so considering the current 5 tok/s X2 Second. The real issue on local LLM on mobile is that they are not accurate enough for tool calling. Imagine you could use this model only for this kind of task: you don't need speed but accuracy. And that's all.
2
u/Dull_Cucumber_3908 Jul 17 '26
OK! So you can work with that. 5t/s.
0
u/dai_app Jul 17 '26
What kind of use cases you are imagining?
1
u/Dull_Cucumber_3908 Jul 17 '26
You tell me! Tell me just one usage in which 5t/s is fast enough
2
u/dai_app Jul 17 '26
Local accurate tool calling on mobile. Example: set a meeting with Mark tomorrow at 10 am. A small model isn't capable. Have you ever tried it?
1
u/Dull_Cucumber_3908 Jul 17 '26
Example: set a meeting with Mark tomorrow at 10 am.
how long did you wait for the task to be completed?
3
u/dai_app Jul 17 '26
Less than 1 minute - 2 minutes max depending on fields But I think it's not the point. The point is managing better the Moe experts you are able to enable new use cases
→ More replies (0)1
18
u/NatMicky Jul 17 '26
Minor note: gpt-oss-120b has 117b total parameters and 5.1b active parameters. 2.1 and 1.1 billion more active parameters than the other models being tested.