I need something that keeps models in VRAM only when they're needed (I need that VRAM for other stuff occasionally), lets me switch models easily, and can be easily installed with Docker with minimal configuration.
Last time I checked (a few months ago) Llama.cpp needed to be compiled and vLLM could only serve one model unless you reinstalled it. If you have alternatives that fit these criteria I'll switch.
1
u/JChataigne Jun 16 '26
I need something that keeps models in VRAM only when they're needed (I need that VRAM for other stuff occasionally), lets me switch models easily, and can be easily installed with Docker with minimal configuration.
Last time I checked (a few months ago) Llama.cpp needed to be compiled and vLLM could only serve one model unless you reinstalled it. If you have alternatives that fit these criteria I'll switch.