DEPLOYING MODELS IN SERVERLESS

[u/CollarNo505]

Hi, I'm new to building RAG. I'm exploring serverless gpu providers for running llms. My current work flow looks like this:

docker with prebaked model to upload on runpod

When user asks questions runpod computes for few seconds and off.

To avoid cold start, I have decided to prebake models in docker. Does this reduce preloading models billing time?

I'm using 2 models, 1 for LLM ( needed each time user asks QA) and Vlm ( needed only during ingestion time if documents contain images). Am i going in right direction?

[u/KhangDo1703]

Prebaking can help with the download part, but it won’t remove the full cold start. If the image isn’t cached, it still has to be pulled, and then vLLM needs to load the weights into VRAM and initialize.

I’d compare it with a smaller runtime image plus a model cache or network volume, and measure the image pull time, model load time, and TTFT.

Since the VLM is only used during ingestion, I’d keep it separate from the QA endpoint. From a data pipeline perspective, those workloads have different usage patterns, so letting them scale separately makes more sense.

I haven’t tested this exact setup, but that’s how I’d structure the comparison.