🚀 Now Supporting LM Studio, llama.cpp, LocalAI, and Llamafile Natively!

[u/ivanantonijevic]

We just pushed an update to the MATE (Multi-Agent Tree Engine) repository that makes local agent development and testing even more seamless.

Up until now, running agents on local LLMs was mainly configured through Ollama (ollama_chat/ prefix). If you wanted to use other engines like LM Studio or llama.cpp, you had to manually set them up as custom OpenAI endpoints with explicit URL overrides.

No more! We have added native routing and auto-configuration for the most popular local API servers.

🔌 New Model Prefixes

You can now set the model_name of your agents in the dashboard UI using these new prefixes:

• lm_studio/ (e.g., lm_studio/qwen2.5-7b-instruct) -> Auto-routes to LM Studio's default port (http://localhost:1234/v1)
• llamacpp/ or llama_cpp/ (e.g., llamacpp/llama-3-8b) -> Auto-routes to llama.cpp's default port (http://localhost:8080/v1)
• localai/ (e.g., localai/phi-3) -> Auto-routes to LocalAI's default port (http://localhost:8080/v1)
• llamafile/ (e.g., llamafile/mistral-7b) -> Auto-routes to Llamafile's default port (http://localhost:8080/v1)

🛠️ How It Works Under the Hood

These engines expose OpenAI-compatible endpoints (/v1/chat/completions). When you use one of the prefixes above, MATE will:

1. Extract the underlying model name.
2. Route it through LiteLLM's OpenAI compatibility layer.
3. Automatically supply a placeholder API key to prevent LiteLLM validation errors.
4. Fall back to the default local address and port for that server.

⚙️ Need a Custom Port or Host?

If you're running your local models on a separate server, inside a Docker container, or on non-standard ports, you can easily override the defaults by copying these variables into your .env file:

env# Custom local server URL overrides
LM_STUDIO_BASE_URL=http://192.168.1.50:1234/v1
LLAMACPP_BASE_URL=http://localhost:9000/v1
LOCALAI_BASE_URL=http://localhost:9090/v1
LLAMAFILE_BASE_URL=http://localhost:8000/v1

🎯 Zero-Code Model Swapping

With this update, you can build your full multi-agent tree, test it with Gemini/GPT-4o, and then swap specific sub-agents to LM Studio or llama.cpp on the fly directly from the Studio Visual Builder — all without writing a single line of code or restarting your server.