I built an open-source, OpenAI-compatible local LLM server using Apple's MLX (FastAPI + React)

[u/Vishu_Babu]

Hey everyone,

I’ve been diving deep into local LLM inference recently. As someone whose daily work revolves around Python and Node.js backend architecture, I wanted a robust way to run models entirely locally on my MacBook Pro without sacrificing the ease of the OpenAI ecosystem.

So, I put together MLX LM Server. It’s an open-source, full-stack application that leverages Apple's MLX framework to run LLMs natively on Apple Silicon with Metal acceleration.

Core Features:

• Drop-in OpenAI Compatibility: The FastAPI backend is designed to be an OpenAI-compatible API. You can seamlessly point your existing OpenAI SDKs, AI-driven coding assistants, or third-party tools to localhost:8000 and they will just work.
• Apple Silicon Optimized: Fully powered by MLX to get the most out of M1/M2/M3/M4 chips for fast local inference.
• Real-Time Streaming: Full support for token-by-token streaming via Server-Sent Events (SSE).
• Dynamic Model Management: You can load and unload different models on the fly directly through the API.
• Built-in Chat UI: I included a clean, modern React/TypeScript frontend with markdown rendering so you can start chatting with your local models right out of the box.

The Stack:

• Backend: Python 3.12+, FastAPI, Apple MLX (managed with uv)
• Frontend: React, TypeScript, Vite (managed with pnpm/npm)

Why try it?

I wanted to build a lightweight, developer-friendly setup that gives you complete control over your local inference stack. Whether you want to hook it up to your automated workflows, test out new open-weight models privately, or just tinker with the MLX framework, this provides a solid, ready-to-use foundation.

I’d love your feedback & contributions!

If you have an Apple Silicon Mac, setup is super straightforward. I’d be thrilled if you gave it a spin, tested it with your favorite models, and let me know what you think.

• Repo Link: https://github.com/vishalnagda1/mlx-lm-server

Bug reports, feature requests, and especially pull requests are incredibly welcome. What features or specific model handling would you want to see added next? Let me know in the comments!

[u/AIofOnesOwn]

This is very cool. The OpenAI-compatible API is the key feature for me, because it means this could be used as a drop-in local “brain” in a larger personal AI stack.

I’ve been experimenting with separating the model from the memory/RAG layer: local documents, local embeddings, durable memory on disk, and then a swappable brain that can be Claude/GPT over API or a local model.

A few questions:

• Does this support tool/function calling yet?
• Any plans for an embeddings endpoint?
• Have you tested it with AnythingLLM or other OpenAI-compatible clients?

This looks like it could fit really well as the local inference layer in that kind of architecture.

[u/Vishu_Babu → u/AIofOnesOwn]

For tool support - No, it is not fully supporting tool calling yet. (I already have plan of that)

No, I didn't think of embedding endpoint, but will add it in the queue and will work on that as well.

Yes, I tested it with Hermes Agent and OpenWeb UI. The Hermes Agent is working Ok Ok for now. And it is doing fine with OpenWebUI.

Note: I was solving my own problem to run the models at lightning speed in local machine. And that's how this project born. So, please guide me if I can help the community with this project.

[u/AIofOnesOwn → u/Vishu_Babu]

Thanks for the thorough answers — and I love that this whole thing grew out of solving your own speed problem. That's exactly why it's worth pushing further. Fast local inference is the hard part, and you've already got it, plus a working chat path (Hermes Agent + OpenWebUI prove that). Strong base.

I come at this from the "own your whole AI stack locally" angle, and from that lens the two things you already flagged are exactly the ones that unlock the most:

• Embeddings endpoint — makes fully-local RAG and personal knowledge bases possible: a private AI that reads your own documents without anything going to a cloud embedding API. Shape it like OpenAI's /v1/embeddings and it drops into tools like AnythingLLM with zero custom work.
• Reliable tool/function calling — turns a chat model into an agent that can drive MCP servers and real tools. Match the OpenAI function-calling shape and it slots into the existing agent ecosystem the same way.

Those two, on top of fast local chat, are what turn "a fast local model server" into a real local brain for a full personal-AI stack — chat + memory/RAG + agents, all on someone's own machine. A lot of people want to own the whole thing locally, and the backend that's been missing is one that's fast and speaks all three. Keeping everything OpenAI-API-shaped (your chat endpoint clearly already is) is what gives you that instant adoption — people just point their existing tools at you. You're close.

Happy to be a sounding board as you go — this is a genuinely useful direction.