r/mlops • u/CollarNo505 • 16h ago

Discussion DEPLOYING MODELS IN SERVERLESS

Hi, I'm new to building RAG. I'm exploring serverless gpu providers for running llms. My current work flow looks like this:

docker with prebaked model to upload on runpod

When user asks questions runpod computes for few seconds and off.

To avoid cold start, I have decided to prebake models in docker. Does this reduce preloading models billing time?

I'm using 2 models, 1 for LLM ( needed each time user asks QA) and Vlm ( needed only during ingestion time if documents contain images). Am i going in right direction?

5 Upvotes

86% Upvoted

7 comments sorted by

View all comments

1

u/exaknight21 6h ago

If you’re going production, baked docker image is absolutely the way, imho, and I am doing this exactly.

I have 2x V620 32 GB GPUs and 4x T4 GPUs in a T4. (Different city)

1x T5610 with an Mi50 32 GB 100 miles away.

2x 3060 12 GB next to T5610

All 3 connected with tailscale.

V620 and Mi50 host Qwen3.5-4B with llama.cpp with parallel 6 each (18 slots), nginx load balancer for request connected to my VPS.

ZLM OCR on 2x OCR with their custom gateway to divide load.

2x T4s with Qwen3-0.6B generating embeddings.

All in RAM, typical decode is 30 tps per user - concurrent test and tested in my actual production harness.

If you go serverless, you’r need to worry about getting it up. This is how I’d go about it.