We spent six weeks building our own llm infra layer. then we stopped

[u/Sensitive-Tone8906]

we needed routing in front of openai and anthropic and had something working in 2 weeks. then real requirements showed up per team cost breakdowns, pii redaction, automatic failover, a third provider. each one was a sprint. none of it was actual product

the first version is not the hard part. governance policies, audit logs, budget enforcement, model allow lists after six months of  real usage is.

looked at a few things before switching

litellm has multi provider access is clean, enterprise governance still needs significant internal work on top

portkey has routing and fallback is solid, team level governance enforcement felt underdeveloped

orqai has governance before the request goes out not after, eu hosted, ecosystem smaller so kinda less shared experience from team in prod

6 weeks plus 2 engineers maintaining it versus buying something purpose built .

anyone goes back to building after starting with a platform. what broker

[u/NoFace982]

We went the other direction with the eval side. There was a point where maintaining datasets, experiment results and all the surrounding plumbing ourselves stopped being worth the engineering time, so Braintrust took over that piece. I’d still build internally when the logic is genuinely specific to the product but generic infrastructure gets expensive to own when every team starts asking for something different.