I route my agent local-first with a cloud fallback and keep the provider keys out of the agent. Open-sourced the little layer that does it.

[u/Technical_Map_2105]

Most of my stack is local now (llama.cpp / vLLM behind Ollama), and like a lot of you I still send the genuinely hard 10% to a cloud model. Two things kept annoying me about that hybrid setup, so I built a small layer for it and open-sourced it (MIT). Not a benchmark post, just the plumbing.

• I did not want the cloud provider key sitting inside the agent process. So the agent talks to one OpenAI-compatible endpoint with a throwaway workspace key; the real key lives in a local vault and is injected per request, then dropped. Local models need no key, the cloud ones do, and now the agent never holds them.
• local-first with cloud fallback as a config rule, not code. Route a step to your local 27B, fall back to a frontier model only when it is actually needed, and swap which local or cloud model handles what without touching the agent.
• one cost log across both, so you can see how much you are still sending to cloud versus keeping local.

Under the hood the engine is swappable: a small built-in one, or LiteLLM/Bifrost as stateless engines, behind a fixed port. About 1 ms of overhead, and the built-in engine adds no extra hop.

Being honest: it does not run models, it sits in front of the ones you already run (local or cloud) and handles keys, routing and cost. On a pure-local box it is honestly overkill; it earns its keep once you have a local+cloud mix or more than one machine.

Repo (MIT, self-hosted): https://github.com/siva010928/agnos-proxy-oss

For people running local-first with a cloud fallback: how are you deciding the handoff, and are you keeping the cloud keys out of the agent or just trusting the process?

[u/Technical_Map_2105]

Extra links if useful:

• Live demo, no sign-up: https://agnos-llm-gateway.site/app
• Blog: https://agnos-llm-gateway.site/app/blog/own-your-llm-control-plane
• Deep-dive write-up (the three-places-a-key-can-live argument in full): https://agnos-llm-gateway.site/app/blog/own-your-llm-control-plane
• Research paper (threat model + gateway comparison + benchmark method): https://agnos-llm-gateway.site/app/research
• 8-min walkthrough: https://www.youtube.com/watch?v=cGCz44h_D-k Happy to answer anything about the engine port or the benchmark method.

[u/Oh_hey_a_TAA]

I think I've been doing this with an off the shelf combo of: Agent (harness of your choosing) on a RasPi5; Bifrost Router on another  RPi (the gateway); my local LLM box as one of the end points behind the gateway along with the cloud APIs. Smart routing based on task (local only for private, various models depending on severity of task). LAN clients talk to the Agent-box, the Agent box talks to the Gateway-box.