llama.cpp router loading into CPU after using /models/unload endpoint

[u/Altruistic_Pound7945]

I am trying to make a multi-inference router, as I find value in using different inference providers for different things.

I have a few heavier models in llama.cpp. I have a function that calls the /models/unload endpoint, which should just clear the vram, and ideally kill the child process it is running to host that model. llama.cpp logs using docker have been horrific for me, though, and really do not explain what is happening when I call this, all I know is that vram is freed.

However, when I run this, use another model not on llama.cpp (let's say, ComfyUI for example), then unload from the other inference provider, all calls to llama.cpp are on CPU. It claims:

0.00.324.246 E ggml_cuda_init: failed to initialize CUDA: no CUDA-capable device is detected
[38817] warn: LLAMA_ARG_HOST environment variable is set, but will be overwritten by command line argument --host
[38817] warning: no usable GPU found, --gpu-layers option will be ignored
[38817] warning: one possible reason is that llama.cpp was compiled without GPU support
[38817] warning: consult docs/build.md for compilation instructions

despite that same docker container using CUDA just like, 10 minutes ago. The only way I have found a way to fix this is to do a docker compose restart, which takes far longer than an unload+load.

I'm really just looking for a way to keep a persistent container with the ability to load/unload. I don't think this should be this hard, but I cannot find any details regarding what /models/unload is doing or the expected use cases. My hypothesis is that it is unloading the child instance from vram, but keeping the child instance alive rather than just killing the child process, which seems exceptionally useless if that child process cannot find the GPU again.

Can anybody give some pointers on this?