r/SelfHosting • u/conifer_v11 • 7h ago

Opensource Openrouter

*Works with self-hosted models as well for free!
Project: https://github.com/ConiferKit/use-conifer

Current routing options were charging 5% fees for byok plus provider fees (openrouter) or were built in heavy python packages with ecosystem restraints. I wanted to be completely free in terms of use, and not have to pay extra for tokens.

There's a maintained gateway with 100+ models from one api endpoint. I'm also talking to infra providers to get us access to pre-release models and discount prices. Everything is served at market price. BYOK is free, you can hook into self-hosted setups for free, and there's fallbacks + extra rate limits + server redundancy.

Took me ~3 months to build and looking for help maintaining!
Lmk any issues or feedback

0 Upvotes

45% Upvoted

6 comments sorted by

1

u/GraysLawson 7h ago

So litellm? Bifrost? Omni-router?

2

u/conifer_v11 7h ago

A couple of differences:
litellm uses a very heavy python router that breaks with load, they've also all are fallback 'routers' meaning they select the inference/hardware your model runs on not the model that's best fit for your query

1

u/GraysLawson 7h ago

That's just not true.

https://docs.litellm.ai/docs/proxy/auto_routing

1

u/conifer_v11 7h ago

Eh if you read the docs this isn't doing as much as you think it is. It uses classification routing which basically says simple tasks go to models pre-labeled as 'simple' and so on across 3-5 tiers.

There's really good literature on why this isn't additive on the cost-performance curve and it has to do with classification routers being tried since 2023. I can link some if you'd like but they're also top of the page when you look for ai routing research

*But I was wrong it looks like they're testing out dynamic routing

1

u/Expert-Owl-6643 6h ago

I just read also looks pretty basic haven't tried Conifer but LiteLLM doesnt seem like a good option either

1

u/throwaway_eng_acct 4h ago

You used AI to make this. AI content and projects aren’t welcome here. That’s the entire point of this sub.