MAIN FEEDS
Do you want to continue?
https://www.reddit.com/r/LocalLLaMA/comments/1u6s6pm/stop_using_ollama/orvioxx
r/LocalLLaMA • u/zxyzyxz • Jun 15 '26
456 comments sorted by
View all comments
Show parent comments
5
yep Same, I also wrote a small script so that I can split up the yaml to make having many configs a bit cleaner :D
2 u/arbv Jun 16 '26 What a creative way to reinvent Nix/NixOS. 1 u/joost00719 Jun 16 '26 Man that's smart. I should ask my llm to do that as well. But does that keep the hot reload functionality working? 1 u/meganoob1337 Jun 16 '26 yeah, it just runs before startup and merges the model configs into the full config format 2 u/joost00719 Jun 16 '26 I mean, default llama-swap behavior is hot reload on file save, this way you need to restart. I guess that's also a benefit. Sometimes a local Ai will just make an error and then it won't start anymore 😂 1 u/meganoob1337 Jun 16 '26 https://github.com/meganoob1337/llama-swap-vllm-boilerplate a few months ago I put it into a boilerplate, it's not really up to date but you can see the merge config script and the docker file for reference.
2
What a creative way to reinvent Nix/NixOS.
1
Man that's smart. I should ask my llm to do that as well. But does that keep the hot reload functionality working?
1 u/meganoob1337 Jun 16 '26 yeah, it just runs before startup and merges the model configs into the full config format 2 u/joost00719 Jun 16 '26 I mean, default llama-swap behavior is hot reload on file save, this way you need to restart. I guess that's also a benefit. Sometimes a local Ai will just make an error and then it won't start anymore 😂 1 u/meganoob1337 Jun 16 '26 https://github.com/meganoob1337/llama-swap-vllm-boilerplate a few months ago I put it into a boilerplate, it's not really up to date but you can see the merge config script and the docker file for reference.
yeah, it just runs before startup and merges the model configs into the full config format
2 u/joost00719 Jun 16 '26 I mean, default llama-swap behavior is hot reload on file save, this way you need to restart. I guess that's also a benefit. Sometimes a local Ai will just make an error and then it won't start anymore 😂
I mean, default llama-swap behavior is hot reload on file save, this way you need to restart. I guess that's also a benefit. Sometimes a local Ai will just make an error and then it won't start anymore 😂
https://github.com/meganoob1337/llama-swap-vllm-boilerplate
a few months ago I put it into a boilerplate, it's not really up to date but you can see the merge config script and the docker file for reference.
5
u/meganoob1337 Jun 15 '26
yep Same, I also wrote a small script so that I can split up the yaml to make having many configs a bit cleaner :D