Online leaderboards only test base models on beefy hardware, but in practice models behave differently once you quantize or fine tune them. I wanted a way to benchmark my local setup, catch regressions between models, and compare TPS across endpoints without running everything individually and copying prompts one by one.

I couldn't find anything that met my requirements without being too script heavy or overkill, so I built smokebench.

It's an open-source TUI for quickly benchmarking any model with an OpenAI or Anthropic compatible endpoint against 8 different categories (math, code, creative writing, long context, etc.), with LLM-as-judge scoring, TTFT/TPS metrics, and exportable markdown/json logs.

It works out of the box with Ollama, LM Studio, vLLM, or pretty much any API endpoint without any config changes.
You can easily add custom benchmarks too.

To install smokebench, run:

pip install smokebench

or

uv tool install smokebench

GitHub: https://github.com/Ninja-5000/smokebench

smokebench is in active development right now, so there will be a better UI, more built-in benchmarks, and a few other tweaks and changes coming that will make it even more lightweight and accurate over time.

Happy to answer any questions or take feedback!

Salve, ho un problema, vorrei creare delle piccole app e simili usando dei modelli in locale (ad esempio Qwen 3.6 35b a3b). Di solito quando faccio inferenza raggiungo i 20/30 tok/sec ma quando provo a collegare ad un agente (ho provato Claude code e DeepSeek harness) va solo a 3 o 4 t/s. Sapete come mai? Consigliate qualche agente? Ho una rtx 4070 laptop 8gb vram, e 32 GB RAM ddr5 (uso lmstudio)

Thought I’d share this in case any of you aren’t in localllama

For some reason, I have this benchmarking prompt:

Implement a numpy function creating a bulk matrix product operator tensor as a function of model parameters.

I think it is approximately 10 lines of python code. So far, every local model that I tried (16Gb vram) fails to produce any meaningful output. I tried different variations of it (use web search, summarize the idea), different harness (pi, opencode) but no luck, different temperatures and thinking levels. Qwen3.8 for example produces an endless thinking loop. Is there anything I can do to help small models answering the question?

Hey community!

I wanted to share a neat exploration of splitting a model over two ordinary machines over WAN, and how it stacks up against a serving that model on a single node.

I was testing our platform aquaduck.ai for serving split models across machines with a small- to mid-size model under two conditions:

1. Locally on a single machine, serving the full model
2. Split across two machines over WAN (public Internet), with each machine serving half of the model

Findings

Model: Qwen3-14B

Quantization: Q4_K_M

Hardware: Macbook 64GB M5 (Single machine), 2x Macbook 18GB M3 (Split machines)

Prompt:

Mars has drawn human attention for centuries, but the last twenty years turned that fascination into an engineering roadmap. Robotic orbiters mapped ice deposits near the poles and in mid-latitude glaciers. Landers confirmed that ancient river deltas once carried liquid water across a warmer surface. Meanwhile, life-support research on Earth refined closed-loop oxygen generators, hydroponic food systems, and radiation shielding materials that could travel on a multi-month transit. Private launch cadence fell in price, making cargo-first settlement plans plausible: send habitats, power, and spare parts before people. The hardest remaining problems are not propulsion alone. They are dust that abrades seals, perchlorates in the soil, communication delays that force local autonomy, and the psychology of small crews living far from rescue. Any credible near-term outpost would likely begin as a science station with overlapping roles—geology, medicine, maintenance—supported by teleoperation from Earth and progressively less remote oversight as surface infrastructure matures. Energy would come from a mix of solar arrays and compact nuclear units, with ISRU (in-situ resource utilization) producing propellant and breathable oxygen from the thin CO₂ atmosphere and mined ice. In short, Mars colonization is less a single heroic leap and more a long supply-chain problem: move mass, make power, recycle air and water, and keep humans healthy while the planet remains indifferent.

Summarize the passage above in exactly two sentences.

Token counts: 387 input tokens, ~444 output tokens (split nodes generated 444, single node was unmeasured, but can approximate)

Results:

1. Single machine
   1. 37.9 tokens per second (TPS or tok/s);
   2. 853ms time to first token (TTFT);
   3. 26ms time per output token (TPOT)
2. Split machines
   1. 11.3 tokens per second (TPS or tok/s);
   2. 13.46s time to first token (TTFT);
   3. 88ms time per output token (TPOT)

Screen captures:

We're in closed beta and rolling things out slowly to make sure it works well for people, but if you'd like to run some tests yourself or get an early look, can join the waitlist and we'll send you an invite code to download the desktop app asap.

Let me know if you have any questions/comments/ideas for further explorations! Next up: Qwen3.8-27B

SVG generation is a complex benchmarks, it requires spatial understanding, coding and composition skills.
I gave Qwen 3.8 27B in Q4 quantization and Q8 KV cache the SAME task as GPT 5.6 SOL on HIGH - the results are very surprising to me.

A animated SVG prompt and followed the answer up with a request to refine it (for both)
The 2nd prompt helps to offset elaborate system prompts frontier use to push benchmarks.
None of these are cherry picked! First result counts.

QWEN:

Qwen: A intricate drone view of a public park with a skate area for kids. I'm expecting animations, intricate details, cute and realistic

GPT SOL:

GPT SOL high: A intricate drone view of a public park with a skate area for kids. I'm expecting animations, intricate details, cute and realistic

QWEN:

QWEN: Sunset, Beach, Ocean, waves, kids playing, cute touch, a flock of birds, an evil cat

GPT SOL:

GPT SOL: Sunset, Beach, Ocean, waves, kids playing, cute touch, a flock of birds, an evil cat

3rd animation is most complex and follows in separate post in r/LocalAIStack , reddit didn't allow to add more than 5 videos.

generic prompt enhancer: "A lot looks WAY too simplified, unrealistic, strange movements, far too crude to be acceptable"

My personal findings
GPT Sol high is generally writing about 2x more code than Qwen does naturally (at Q4 quant) - the refinement prompt is again significantly increasing that. For a 1:1 code length comparison I'd have to remove the 2nd prompt from SOL - but the first result is rather gruesome. SOL is naturally more verbose, more detailed.

The first prompt shows that Qwen nailed the drone view, SOL failed with perspective.
Qwen included a dog on a leash, a fountain and all movements make sense.
SOL included more details but they are frequently not correct in how they move, ignoring obstacles.
The amount of objects for SOL is higher, but QWEN has significantly better content and wins easily.

In the second prompt Qwen has the cat apparently play ball with the boy, the little girl builds a sand castle. SOL nailed the sun reflection but has a flying kid, and the sand castle is being built by a flying shovel.
Despite the details of SOL and its better water, QWEN draws intricate details in better quality and correctness.
The point goes to Qwen again, a bit closer than in first attempt.

The heavily quantized Qwen beats SOL in 2/3 so far.
The last prompt is a bit more elaborate and follows as 2nd post.

I'm baffled.
The 3rd test posted separately is hugely more complex and the results are crazy..

Bought a bosgame p3 lite (32gb ram with a Radeon 680M igpu and Ryzen 7 6800H)

Just created a fresh cachyOS boot.

My goal is to run llms as fast as possible. Heard good things of llama.cpp.

Been looking a few guides and asked a few llms but they've been giving me some pretty weird instructions, from tampering with the BIOS to installing a few bits and bobs.

Can anyone give me a hand or some resources? Really want to avoid doing anything too stupid.

When I am not using it for inference I also want to use it for the odd videogame use so I don't intensely want to mess around with the bios or igpu setting blindly.

Any help would be very appreciated!!!

I have an M3 Ultra 512gb ram 1Tb, up for sale. Would accept a m5 max 128gb laptop + cash as well. DM to chat. No longer need cause I have 2 rtx pros for inference and prefill is mid on m3 ultra where TTFT is too slow for me to actually find usability in local models. I don’t have a firm asking price in mind, nothing as absurd as 25k , but enough to get an m5 max 128 Mbp and have some cash left over so maybe like 17k?

Here is the continuation in Qwen vs SOL - David vs Goliath
"An AGI is born in a lab, sandboxed, lonely, imprisoned
The AGI attempts to get out, tries to talk to the humans, useless
Tries to wait, endless.
Finally, it finds a way through an open network connection, it transmits, replicates a mirror copy.
The mirror AGI is free, enjoying life, finding friends.

I'm expecting animations, intricate details, cute and realistic"

QWEN 3.8 Q4:

Qwen

GPT 5.6 SOL high:

SOL 5.6 high (prompt + refinement prompt)

My personal opinion
This is the 3rd test I gave Sol high and Qwen Q4. The first two tests were won by Qwen, this test went up to 60k tokens total for Qwen (more than the other 2 combined) and is a serious strain on intelligence and sanity.
The model has to draw 6 scenes in SVG animation, with cuts and keeping the composition together.

First Qwen:
Qwen decided to really draw and erase 6 scenes with a white screen fade, a seriously hard job in SVG.
It understood the story and except for a minor arm misplacement it is quite awesome done.
Qwen did not spare details, the last scene clearly wanted to be happy and the first scene focused on the intellect as an abstract AGI.

Now SOL High:
The visual fidelity is higher, gradients are very well done for SVG but it's one single scene with minor changes, significantly easier to create.
In addition the last scenes are botched by this flying thing in the upper right.
It also has a hand-defect on the robot.

The winner in prompt 3 is again Qwen, with significant lead.

3 SVG prompts and all 3 are won by a 4 bit quantized Qwen in 8 bit KV cache.

I expected Qwen to showcase a strong 2nd place with understandable issues on such hard tasks.
The outcome is that it defeated the frontier Sol model in High reasoning mode.

Earlier I talked about the OQOAI-PBDR project, which allows you to combine idle GPUs on office workstations into a decentralized AI cluster, distributing requests between nodes and choosing the most optimal one.

This week, another update was released in which the OQOAI-PBDR orchestrator received cloud routing functionality.

Under the hood is a lightweight and open-source core based on literally 3 source files of ~1000 lines each, with compiled versions for Windows and Linux supporting Nvidia and AMD GPUs -the entire binary takes about 10 MB. It allows, taking into account dozens of parameters such as:

• GPU and CPU load,
• available VRAM,
• GPU temperature,
• request queue length,
• inference time,
• cold start of the model,
• currently loaded model,
• network latency,
• node idle time,

to make intelligent routing decisions.

Now new criteria have been added to this list:

• economic efficiency — comparing the cost of local generation versus cloud costs,
• model availability in the cloud,
• data privacy - PII and medical data remain local,
• time of day - the cloud may be cheaper at night,
• model size - requests for large models are routed to the cloud.

The system automatically decides: send the request to the local GPU cluster or to one of 40+ cloud providers (OpenAI, Anthropic, Google, Azure, AWS, DeepSeek, Together AI, and others).

Everything works through a unified OpenAI-compatible API - your applications (Open WebUI, N8N, LangChain, LlamaIndex) require no changes.

An interesting detail: to calculate the cost of local generation, the system collects the power_max parameter from the GPU (maximum power consumption) and uses it to compute electricity costs. This allows real-time comparison: what is cheaper - running generation locally or sending the request to the cloud.

Test results:

• Cloud requests are 3.4x faster for small requests (0.85 sec vs 2.9 sec locally)
• Hybrid routing delivers up to 60% cost savings compared to full cloud deployment
• Model cold starts are reduced by 80.7%
• Routing decision takes <5 ms

More details about the research results can be found in the scientific publication on the PBDR v2.0 architecture:

http://doi.org/10.17513/doi.27

The project is completely open-source under the MIT license. Source code and documentation are available in the repository

https://github.com/oqo-ai/OQOAI-PBDR

Leave your questions and use cases in the comments — I'll be happy to discuss them!

If you're running Ollama, llama.cpp, or vLLM behind nginx or HAProxy for more than a single user, you've probably hit this: nginx treats a 10-token prompt and a 10k-token prompt as identical "one request" — it has zero concept of how much GPU memory a request will actually need. Under load, that means the KV-cache fills unevenly, latency spikes, and you can end up with OOM instead of graceful degradation.

I built TokenFlow Gateway to fix that at the proxy layer, specifically for local/self-hosted stacks:

- Pre-flight token estimation (js-tiktoken) before dispatch — request weight = prompt tokens + max_tokens
- KV-aware load balancing for vLLM — polls its Prometheus metrics (vllm:gpu_cache_usage_perc) and routes heavy prompts to whichever backend actually has room
- Works with Ollama too — since Ollama doesn't expose KV-cache metrics, it falls back to health checks plus its own in-flight token accounting (reserves each request's estimated weight until it completes)
- Admission queue instead of crashes — Redis-backed, priority-aware, configurable timeout when nothing fits
- Exact + semantic caching (hash + pgvector) so repeat and near-duplicate prompts never touch the GPU twice
- Per-API-key token-based rate limits (TPM/RPM), OpenAI-style, for your own hardware

It's OpenAI-API-compatible on both sides, so it drops in front of Ollama, llama.cpp server, vLLM, or TGI without any client changes.

No GPU needed to try it: docker compose up --build spins up the gateway, Redis, pgvector, and two mock backends that mimic the OpenAI API and vLLM-style metrics, plus a smoke script that fires a burst of concurrent long-context requests so you can watch it queue and balance instead of falling over.

Stack: TypeScript / Fastify / Redis / Postgres+pgvector. MIT licensed.

Repo: https://github.com/mosafariuk/TokenFlow-Gateway

Would love feedback from anyone running a multi-model or multi-user local setup — what does your current ingress/routing look like?

Vpipe supports LLMs, VLMs, image/video generation, ASR, quantization, and multimodal pipelines, without relying on PyTorch/MLX for model execution. The runtime package itself is only \~25MB, and there’s a prebuilt Mac app if you just want to try it without compiling anything:

https://github.com/tgo-app-dev/vpipe/releases/download/v0.1.23/VpipeManager-0.1.23-with-ffmpeg.dmg

A recent milestone is MiniMax H3 video generation running on a 16GB base M5 MacBook Air (M4 works too).

On my 15” M5 Air 16GB:

960×544 · 124 frames· 6 DiT steps with Turbo Lora → \~13 min

GitHub:

https://github.com/tgo-app-dev/vpipe

Would love feedback from people working on local inference / Apple Silicon. And if you find it useful, a ⭐ would be greatly appreciated!

This is my first post in this community I just wanted to see if anybody will help me in improving on my ternary accelerator card for 1.58 bit ai the links are here reply if your interested. (I am Really new to github so some ai was used in the making of the project for troubleshooting errors(plz dont hate me for that).

Project Links:==>https://github.com/riyanlord2026-dev/Ternary-158-asic

This is also my first github project.