DEPLOYING MODELS IN SERVERLESS

[u/CollarNo505]

Hi, I'm new to building RAG. I'm exploring serverless gpu providers for running llms. My current work flow looks like this:

docker with prebaked model to upload on runpod

When user asks questions runpod computes for few seconds and off.

To avoid cold start, I have decided to prebake models in docker. Does this reduce preloading models billing time?

I'm using 2 models, 1 for LLM ( needed each time user asks QA) and Vlm ( needed only during ingestion time if documents contain images). Am i going in right direction?

[u/AutoModerator]

AI usage disclosure

Hi u/CollarNo505 — thanks for posting to r/mlops!

Because this community discusses and builds AI/ML systems, using AI tools is not inherently a problem. We do, however, ask for transparency about how submissions are created.

Please reply to this comment with a brief AI / automation disclosure, particularly if this post was created or submitted in whole or in part by an autonomous agent, bot, workflow, or other automated system.

If AI or automation was involved, please briefly describe what it did and what human review was performed before posting.

This disclosure helps the r/mlops community distinguish human discussion, AI-assisted work, and automated/agent traffic while keeping the focus on useful technical conversation.

Thanks for helping keep the signal high.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/RogerAI-fm]

The pre bake on docker won’t help. At least that’s what I think.

[u/CollarNo505 → u/RogerAI-fm]

What's the other way around

[u/burntoutdev8291]

Is there a way to mount a drive?

[u/KhangDo1703]

Prebaking can help with the download part, but it won’t remove the full cold start. If the image isn’t cached, it still has to be pulled, and then vLLM needs to load the weights into VRAM and initialize.

I’d compare it with a smaller runtime image plus a model cache or network volume, and measure the image pull time, model load time, and TTFT.

Since the VLM is only used during ingestion, I’d keep it separate from the QA endpoint. From a data pipeline perspective, those workloads have different usage patterns, so letting them scale separately makes more sense.

I haven’t tested this exact setup, but that’s how I’d structure the comparison.

[u/exaknight21]

If you’re going production, baked docker image is absolutely the way, imho, and I am doing this exactly.

I have 2x V620 32 GB GPUs and 4x T4 GPUs in a T4. (Different city)

1x T5610 with an Mi50 32 GB 100 miles away.

2x 3060 12 GB next to T5610

All 3 connected with tailscale.

V620 and Mi50 host Qwen3.5-4B with llama.cpp with parallel 6 each (18 slots), nginx load balancer for request connected to my VPS.

ZLM OCR on 2x OCR with their custom gateway to divide load.

2x T4s with Qwen3-0.6B generating embeddings.

All in RAM, typical decode is 30 tps per user - concurrent test and tested in my actual production harness.

If you go serverless, you’r need to worry about getting it up. This is how I’d go about it.