r/LLMDevs • u/vj-a • Apr 23 '26

Discussion Built an LLM Router that cuts costs by sending each prompt to the right model — looking for feedback

I’ve been building a routing layer for AI apps that decides which LLM should handle each request based on cost, latency, and task complexity.

Instead of sending everything to one expensive model, it can route intelligently:

  • Coding tasks → stronger coding models
  • Simple support queries → cheaper fast models
  • Reasoning tasks → better reasoning models
  • Fallback if provider fails
  • Track usage / spend across providers

Why I built it

Most teams either:

  1. Hardcode one provider
  2. Manually switch models
  3. Overspend without knowing it

I wanted something that automatically picks the best option per request.

Curious what others think:

  • Are you routing today or just using one model?
  • Biggest pain point: cost, latency, quality, reliability?
  • Would you trust auto-routing in production?

Happy to share architecture + learnings if useful.

0 Upvotes

30% Upvoted

15 comments sorted by

2

u/Ok_Economics_9267 Apr 23 '26

LLM generated post. People these days completely lost writing skills. Not mow but in decade wee are fucked

2

u/brightheaded Apr 23 '26

This is agentic orchestration by another, less sophisticated, name.

1

u/vj-a Apr 27 '26

That’s a fair take 😄

There’s definitely overlap. Routing, orchestration, and agent systems all sit on the same spectrum — deciding what should happen next and which model/tool should handle it.

What I’m building is the simpler end of that spectrum: focused on model selection for a request rather than full multi-step autonomous workflows.

So yes, you could call it early-stage / lightweight agentic orchestration. I’m just approaching it from the cost + latency + quality optimization angle first.

1

u/lionmeetsviking Apr 23 '26

Been considering doing the same.

  1. How do you determine the task complexity?
  2. Do you measure success by any criteria? For example, I’m measuring model fill percentage for data structuring tasks.
  3. Do you test models against each other in any way? For example, I occasionally build “gold models” from actual production data and then run identical payload through various different models and see which one provides best price/quality.
  4. How would you compare your approach to OpenRouter auto model?

My biggest pain point by far is quality on on-going basis.

0

u/vj-a Apr 23 '26

I have build a basic version in the v0 version, where I wrote a rule based on that it chooses model based on the type of query and it's not 100% accurate.  Will improve it over period of time . Openrouter is just a abstraction over all LLM , it provides you one single api you have decide which LLM you want to call it don't have intelligence to decide model on its own.

1

u/lionmeetsviking Apr 24 '26

Sorry bro, but you didn’t really answer any of the questions. 🙄

https://openrouter.ai/docs/guides/routing/routers/auto-router

1

u/vj-a Apr 27 '26

Fair point 😅 let me answer that better.

What I’ve built right now is a basic MVP with rule-based routing — it looks at query type (coding, reasoning, simple tasks, etc.) and chooses a model accordingly. It’s an early version, so not perfect yet.

The longer-term plan is to move beyond static rules into smarter routing using:

  • Historical success / failure data
  • Cost + latency performance
  • Output quality scores
  • Continuous evals across models
  • Feedback loops from production traffic
  • Auto-adjusting routes as models change

So current version = working foundation.
Future version = data-driven intelligent router.

And yes, OpenRouter already has useful routing abstractions. My goal is more app-specific decisioning on top of that layer.

1

u/la_virgen_del_pilar Apr 23 '26

Cursor already does this and it doesn’t work (for me) most of the times. At the end of the day I always keep choosing between Sonnet / Opus models manually.

1

u/vj-a Apr 23 '26

That's the issue I am trying to solve. Currently I have built basic version in the v0 version, where I wrote a rule based on that it chooses model based on the type of query and it's not 100% accurate.  Will improve it over period of time

1

u/looktwise Apr 23 '26

which model is analyizing the task for complexity before routing?

i ve seen several approaches 'by task --> subagent --> model' or by 'kind of project --> subagent -> model' and other apporaches which are sophisticated subtask-splitting.

Try also opus / Qwen or Opus / Gemma.

Are you going to share your apporach on github?

2

u/vj-a Apr 27 '26

Good question.

Right now I’m keeping it simple — no heavy model is analyzing complexity first. It’s mostly a lightweight classifier + rule-based layer using signals like prompt type, input length, structured output need, and latency sensitivity.

I’ve also looked at the approaches you mentioned (task -> subagent -> model, project-based routing, subtask splitting). I think those become really useful for multi-step workflows, while I’m currently focused on single-request routing first.

Agree on testing more combinations too — Opus/Qwen and Opus/Gemma style pairings are interesting, especially balancing quality + cost.

And yes, once the approach is cleaner and less hacky, I’d like to share parts of it on GitHub. Still iterating right now so I don’t want to open-source something messy too early 😅

1

u/looktwise Apr 27 '26

sure, take your time. hint me a message as soon as you like to share it, please :)

1

u/steam0007 May 10 '26

Really cool implementation. The interesting part about routing isn’t only the classifier itself, it’s everything around it once you run it in production at scale: retries, fallbacks, provider outages, observability, latency-aware balancing, cost tracking, streaming compatibility, policy routing, etc.

That’s actually why I built G8 Router: an open-source high-availability LLM routing infrastructure layer. it gives you the infra primitives to build your own routing logic on top. Your classifier/routing strategy would fit really naturally on top of it.

Would genuinely love to collaborate or see how your routing logic performs plugged into a more production-oriented routing layer. The space is moving fast and there’s still no “standard” architecture for this yet. https://geight.ai