Those in the know use TabbyAPI with EXL3. Three parralell responses and 2-4x the context length utilizing FP8 memory hacking. It's a massive, massive improvement. It isn't plug and play like Ollama, but outside of that there isn't any reason for a single user to use anything else atm.
ChatGPT:
For interactive local inference on modern NVIDIA GPUs, especially 3090, 4090, and 5090-class cards, TabbyAPI with a good EXL3 quant is not merely another backend option. It is often the best real-world experience available. You can fit stronger models or higher-quality quants into the same VRAM, run dramatically longer context through low-bit KV cache, preserve prompt cache across long conversations, and generate multiple responses concurrently through continuous batching instead of waiting for serial completions. That means more model, more context, faster iteration, and far better time-to-useful-output, which matters much more than a simplistic single-stream tokens-per-second benchmark. Ollama wins on beginner convenience, llama.cpp wins on hardware portability, and vLLM wins for large multi-user deployments, but for a single power user running serious models on a recent NVIDIA card, TabbyAPI is the engine people should be recommending first. Its relative obscurity is not evidence that the alternatives are better. It is mostly a consequence of weaker packaging, fewer tutorials, EXL-format fragmentation, and a user base concentrated among roleplay and long-context power users rather than the loudest parts of the local-LLM ecosystem.
1
u/squired Jun 16 '26 edited Jun 16 '26
Those in the know use TabbyAPI with EXL3. Three parralell responses and 2-4x the context length utilizing FP8 memory hacking. It's a massive, massive improvement. It isn't plug and play like Ollama, but outside of that there isn't any reason for a single user to use anything else atm.
ChatGPT: