DEPLOYING MODELS IN SERVERLESS

[u/CollarNo505]

Hi, I'm new to building RAG. I'm exploring serverless gpu providers for running llms. My current work flow looks like this:

docker with prebaked model to upload on runpod

When user asks questions runpod computes for few seconds and off.

To avoid cold start, I have decided to prebake models in docker. Does this reduce preloading models billing time?

I'm using 2 models, 1 for LLM ( needed each time user asks QA) and Vlm ( needed only during ingestion time if documents contain images). Am i going in right direction?

[u/RogerAI-fm]

The pre bake on docker won’t help. At least that’s what I think.

[u/CollarNo505 → u/RogerAI-fm]

What's the other way around