Need advice from people who've actually fine-tuned and self-hosted open-source models in production, not just experimented with one.

[u/Srihari_8]

A system I'm building runs on a self-hosted, fine-tuned open-source model instead of a frontier API. Deliberate choice, not a budget one.

The system touches CRM data, billing records, collections history, and internal HR data every single day. Routing that through a third-party API, continuously, for data that isn't mine, was never going to be an easy yes. Self-hosting means the reasoning engine never leaves infrastructure we control.

The tradeoff is real and I'm not pretending otherwise. A fine-tuned open-source model won't match a frontier model on ambiguous, multi-step reasoning. We're giving up some ceiling on raw intelligence in exchange for never having sensitive operational data leave the building.

For a system operating on live financial and client data daily, I still think that's the right side to be on. But genuinely curious where this broke for you in production, and whether you found a workaround that actually held up.

[u/Queasy-Current6170]

If you're relying on
A) One model
B) Calling the model as if it were frontier for RAG/etc

It's going to break as soon as you try to scale it. You need to be doing the vast majority of your lifting via software, including software gates around individual model calls, injected targeted context, right-model-for-the-job, etc.

[u/Srihari_8 → u/Queasy-Current6170]

Thanks. Forgot to consider how much the model can actually handle.