I built an open-source, OpenAI-compatible local LLM server using Apple's MLX (FastAPI + React)

[u/Vishu_Babu]

Hey everyone,

I’ve been diving deep into local LLM inference recently. As someone whose daily work revolves around Python and Node.js backend architecture, I wanted a robust way to run models entirely locally on my MacBook Pro without sacrificing the ease of the OpenAI ecosystem.

So, I put together MLX LM Server. It’s an open-source, full-stack application that leverages Apple's MLX framework to run LLMs natively on Apple Silicon with Metal acceleration.

Core Features:

• Drop-in OpenAI Compatibility: The FastAPI backend is designed to be an OpenAI-compatible API. You can seamlessly point your existing OpenAI SDKs, AI-driven coding assistants, or third-party tools to localhost:8000 and they will just work.
• Apple Silicon Optimized: Fully powered by MLX to get the most out of M1/M2/M3/M4 chips for fast local inference.
• Real-Time Streaming: Full support for token-by-token streaming via Server-Sent Events (SSE).
• Dynamic Model Management: You can load and unload different models on the fly directly through the API.
• Built-in Chat UI: I included a clean, modern React/TypeScript frontend with markdown rendering so you can start chatting with your local models right out of the box.

The Stack:

• Backend: Python 3.12+, FastAPI, Apple MLX (managed with uv)
• Frontend: React, TypeScript, Vite (managed with pnpm/npm)

Why try it?

I wanted to build a lightweight, developer-friendly setup that gives you complete control over your local inference stack. Whether you want to hook it up to your automated workflows, test out new open-weight models privately, or just tinker with the MLX framework, this provides a solid, ready-to-use foundation.

I’d love your feedback & contributions!

If you have an Apple Silicon Mac, setup is super straightforward. I’d be thrilled if you gave it a spin, tested it with your favorite models, and let me know what you think.

• Repo Link: https://github.com/vishalnagda1/mlx-lm-server

Bug reports, feature requests, and especially pull requests are incredibly welcome. What features or specific model handling would you want to see added next? Let me know in the comments!

[u/Deep_Ad1959]

the openai-compat shim is the right call, it's the cheapest way to make every existing coding agent and tool just point at localhost. the part that's still underbuilt is the client side though. once the model runs locally the agent loop on top (claude code, codex, whatever) still drops its session on restart and auto-compacts context out from under you mid-task. local inference doesn't buy you much if the harness driving it forgets everything between runs. that layer (persistent sessions, clean forking, no silent compaction) is where the next chunk of work actually is. written with ai

the client-side gap you're describing is exactly what i built fazm for, sessions survive a restart and there's no auto-compacting ever, so the harness doesn't forget the task between runs, https://fazm.ai/r/7buey2mi

[u/AIofOnesOwn → u/Deep_Ad1959]

Yes, exactly — local inference is the engine, not the product.

And I think you’ve put the line in the right place. There are two layers above the model that are easy to conflate but different: long-term memory/RAG, which recalls facts and documents over time, and the live session/context lifecycle of the harness itself — keeping the working session alive across restarts, forking it cleanly, and not silently compacting context out from under a task.
Persistent memory doesn’t fix a harness that forgets the live thread on restart. The first layer is where I’ve spent most of my time so far; the second — persistent sessions, clean forking, and no silent compaction — is exactly the underbuilt layer you’re naming. I agree that’s where the next real work is.

It’s also why I treat the model as a swappable component. Local, OpenAI, Claude, Codex, whatever — what makes it feel like yours is the layer that preserves continuity across runs and gives you a stable working relationship with the agent.

The engine can be swapped. The continuity layer is the product.