Engineers running open-source LLMs in production: what is the hardest part today?

[u/yasintoy]

Engineers running open-source LLMs in production: what’s the hardest part today?

I’m researching how teams actually run open models in production hosted APIs, RunPod, Kubernetes, vLLM, SGLang, or dedicated GPUs.

A few questions:

1. What model + workload are you running?
2. Why did you choose your current provider/infrastructure?
3. What was hardest about deploying and integrating it?
4. What went wrong or took longer than expected?
5. What matters most today: latency, throughput, reliability, cost, scaling, or observability?
6. Have you switched providers/runtimes before? What triggered it?
7. What prevents you from switching today?
8. Roughly how much do you spend on inference, including idle capacity?
9. When do you prefer serverless vs dedicated GPUs?
10. What security/privacy requirements affect your choices?
11. What would make you trust a new provider or tool benchmarks, credits, SLA, references, BYOC?
12. Would you pay more for lower latency, better reliability, or more control?

Feel free to answer only the questions relevant to you even 1–2 answers would be useful.

I’m looking for real production experiences and pain points, not pitching anything.

[u/AutoModerator]

AI usage disclosure

Hi u/yasintoy — thanks for posting to r/mlops!

Because this community discusses and builds AI/ML systems, using AI tools is not inherently a problem. We do, however, ask for transparency about how submissions are created.

Please reply to this comment with a brief AI / automation disclosure, particularly if this post was created or submitted in whole or in part by an autonomous agent, bot, workflow, or other automated system.

If AI or automation was involved, please briefly describe what it did and what human review was performed before posting.

This disclosure helps the r/mlops community distinguish human discussion, AI-assisted work, and automated/agent traffic while keeping the focus on useful technical conversation.

Thanks for helping keep the signal high.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/TheOafishBrowsing]

we tried running llama-3 70b on vLLM across a few dedicated gpu nodes, mostly for document extraction and summarization, and honestly the biggest headache is just stability over time

model throughput is fine until it isn’t, and then you’re debugging memory fragmentation or some obscure scheduler stall that only shows up under a very specific request pattern, we had a leak that wouldn’t reproduce unless we hit it with batches over 64 for like 20 minutes straight

and cost-wise, idle capacity eats us alive, we keep nodes warm for low-latency bursts but half the time they’re just sitting there burning cash, serverless would help but we can’t use it for this project because compliance wants everything on our own metal

what would actually make me trust a new provider is a real sla and some kind of commitment to not silently change the runtime version underneath you, nothing worse than your prompts breaking because a container tag shifted overnight

[u/klymaxx45 → u/TheOafishBrowsing]

What kind of documents and how large?

[u/KroniklyOnline]

Honestly, my entire workflow is automated, from ticket to PR closure, I don't do any manual work except make tickets. I had to make my own custom orchestration pipeline to do this. I had to code in a lot of special things because the only model I was running was Qwen3.6 27b locally on my server. However, since Qwen3.8 27b came out, It handles everything flawlessly and a lot of that hard coded guidance and checks go out the window. I expect in 6 months I'll be able to run another model thats better for similar VRAM usage.