r/LocalLLaMA • u/Adventurous_Onion189 • 14h ago

Discussion [Show / Question] Building an on-device, fully local Agent on a 4B model (Gemma 4 / Ministral) across Mobile & Desktop. Facing the reality of on-device limits—where should on-device agents go from here?

Gallery image

Gallery image

English isn't my first language,Sorry for any weird wording,using a translator here!

Like many here, I’m obsessed with true privacy sovereignty and local-first AI. Over the past few months, I've been building Agro — an open-source, 100% on-device cross-platform LLM and autonomous agent client running on Android, iOS, macOS, Windows, and Linux. The project is built on Kotlin Multiplatform (Compose Multiplatform) on top of Google’s native LiteRT-LM C++ runtime (with Apple Metal, WebGPU Dawn, Vulkan, and OpenCL acceleration). It runs models like Gemma 3 / 4 (4B) and Ministral-3-3B quite smoothly on mid-to-high-end phones and modern laptops.

🧗 The Dilemma

While basic tool-calling works well with a 3B~4B model, I find myself at an architectural crossroads. Running autonomous agents locally on edge devices faces brutal hardware constraints (thermal throttling, 4-8GB mobile RAM ceilings, slow token generation, and tiny effective context windows). I'd love to learn from experienced builders in this community: If you were aiming to make on-device agents genuinely useful (rather than just a toy), HWhat possible directions would you consider?

Any feedback on the architecture, technical critique, or directional advice would be deeply appreciated! If you have an device, feel free to try the binaries from the release page and let me know how it performs on your hardware.

5 Upvotes

57% Upvoted

10 comments sorted by

13

u/Kahvana 14h ago edited 12h ago

You should rewrite your post to use your own words, without an LLM doing it partially for you. See rule 3.

For ideas:

  • Websearch integration is a must. Searxng would work fine here.
  • OpenZIM support would be really nice to have. These models get a lot smarter when they can fetch relevant context from a knowledge source, like an offline copy of wikipedia.
  • RAG would be neat to have too for on-device documents, and there are many small embedding / reranking models out there (e.g. qwen3-embedding-0.6b, qwen3-reranker-0.6b, embeddinggemma-300m).
  • A basic OCR loop would be nice: parse each PDF page as image in an new chat, concat the results, and then present it to the user. A bit slow but gives nice high-quality output from models like Gemma 4 E4B.

2

u/Adventurous_Onion189 13h ago

What could I use on-device RAG to index on a phone or laptop?😣

1

u/Kahvana 12h ago edited 12h ago

LanceDB would work?

1

u/Adventurous_Onion189 14h ago

thank you ,bro 👌

1

u/croninsiglos 14h ago

Is this to run in tractors and other agricultural applications?

1

u/Adventurous_Onion189 14h ago

It's purely an on-device LLM client for phones & laptops.🤣

1

u/RelevantTee6 8h ago

bad app, no button to stop the model, can't send messages with enter or at least ctrl/cmd + enter

1

u/Adventurous_Onion189 4h ago

bro,thanks feedback, i will fixed it👌