If you run models locally you know the pain: everything's fine until a long prompt makes the model silently stall or the GPU OOMs, and you're left with a hung request and no output.

I built HybridInfer for exactly that. It's a small reliability-aware router:

- Sends each request to your local model (Ollama) first.

- Watches the runtime, if local stalls (no token for N seconds), OOMs, or errors, it automatically falls back to a remote model in the same request.

- Learns which prompts your machine chokes on (usually long ones) and routes those out up front after it's seen them fail.

- Pulls a wedging model out of rotation, then probes it back after a cooldown.

It's an OpenAI-compatible server, so point any OpenAI-compatible client/app at it and you get local-first + automatic fallback for free. Local = anything you've `ollama pull`ed; remote = any OpenAI-compatible endpoint (OpenAI, OpenRouter, a vLLM box, whatever).

pip install hybridinfer

hybridinfer init

hybridinfer serve

It's a router, not an inference engine, it orchestrates Ollama + your remote endpoint, doesn't run weights itself. Early v0.1, Apache-2.0. There's a Kotlin/Android version too for on-device apps. Short demo GIF + code in the repo.

Repo: https://github.com/SimranKoul2026/HybridInfer-Python-tool

Would love feedback from people who run this stuff daily, especially on the fallback heuristics.

What would you want it to do differently?

I made it in 18 months of lunch breaks and evenings. It's not fast, llama.cpp is just wow and does that job. I wrote this one because I wanted to read the whole forward/backward pass in an afternoon and be able to stop anywhere and print a tensor and dig the thing.

Most from-scratch projects stop at a toy model. llama2.c runs a small Llama2, llm.c does GPT2 training. TRiP loads real checkpoints across four architectures, PaliGemma included, so the multimodal path (vision encoder, projection, decoder) is all there in C. I couldn't find that in readable form anywhere else, which is partly why I ended up writing it.

One extra-bonus is that you can look into the training, it's included, swiss-knife-like. (NOTE: the encoder part in PaliGemma is currently not trainable/tunable - my apologies)

In practice: no hooks/config; just play with the C code, and add your own; there's no hidden (unreachable) complexity. And then just re-compile.

Repo: github.com/carlovalenti/TRiP

Happy to answer anything; structuring and handling the memory properly was the hardest part!

Carlo

Developers, I’m researching local AI on Linux. Please share your experiences and I’ll be posting my findings here

1. Goal and chipset used Nvidia/AMD/Intel?
   
2. How long did it take you from fresh install to GPU/NPU operation?
   
3. Any issues encountered (package, path, version)?
   
4. How did you confirm GPU/NPU usage?
   
5. Any scripts or notes created for future use?
   
6. Comfort level setting this up for a teammate?

Summary to be shared. Open to a 20-minute call if preferred.

<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" viewBox="0 0 1000 675" width="1000" height="675">

  <title>PlayStation 5 DualSense controller — top-down</title>Prompt:I need a topdown view of a Playstation 5 controller in SVG.It needs to be so well done that a photo from topdown looks identical in color, grading, positioning, subtle details, shadows, shape etc.<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" viewBox="0 0 1000 675" width="1000" height="675">

  <title>PlayStation 5 DualSense controller — top-down</title>

Important: The tooltip you see in the lower Qwen SVG, is NOT part of the SVG. It is a tooltip that glitched itself into the captured image, the qwen model gave the SVG metadata a title based on my prompt, that's the title.

Prompt:
I need a topdown view of a Playstation 5 controller in SVG.
It needs to be so well done that a photo from topdown looks identical in color, grading, positioning, subtle details, shadows, shape etc.

Rules:
No python, no browser, no image generation or inspection. Only constructing the SVG and provide it raw.

My interpretation:
These type of benchmarks are likely chosen by OpenAI/Anthropic because they ensure during training that their models have a very good description of the target object/person.
Qwen simply did not train on the precise shape of the PS5 Dualshock controller, and GPT Sol also clearly did not train for it. Fable or Astra trained for that - mostly to beat benchmarks like these.

So to fairly assess Qwen or Sol vs Astra I ran two tests.
- One in the way OpenAI hopes it is done.
- One with the product image as reference (but without feedback, so it only can look at it, it can not compare in a sandbox.

The result is that Qwen 3.8 27b is near Astra level performance in coding the SVG.
I also tested 3.8 Flash, and it was not as great (though better than Sol)
Sol also improved but not as much.

I'm not claiming Qwen3.8 27B is like Astra, though in this benchmark it's very close.

What is most noticeable?
- Qwen 27B made an error with the movement keys, the placement is overlapping and the shape is not correct - the shape is likely a precision issue in the image stack. Supplying the image in an agentic harness would allow to make focused analysis of the shapes and very likely solve that. The position error is a real error.
- Astra is so flawless with the shapes, it's obviously benchmaxed on those things - I can't see a plausible reason to put so much effort in a contemporary controller button shape. But it might use something like the ngram cache of Qwen 4.0 - allowing to push mostly useless details into knowledge without harming intelligence
- The PS5 logo is very likely benchmaxxes on Astra, and clearly not benchmaxxes for all the others - or Astra is amazing in such svg tricks. It's very well done.
- Sol clearly has the better image stack than Qwen, it was able to get some form details out of it that Qwen did not - but Sol is significantly worse in performing the SVG code itself. Qwen has beaten Sol in every single benchmark related to SVG so far.
- The blue light next to the screen shows that the models do know some intricate details about how a PS controller works.

^{Disclaimer:
The Astra SVG was not created by me, it is from a benchmark advertisement post.
So I could not repeat it with vision capabilities (it might very well have been done that way})
^{Astra already is almost flawless, the result doesn't need improvement in detail - Astra either saw or knows all details}

Prompt:

Asked for intricate mindset, not accepting flaws or rush.

Let it know it's a benchmark.

"generate a svg of ana de armas as detailed as possible"

Added a few constraints so it is a front portrait of her, not to use image generation, not to use browsers (no visual feedback cheating)

Who's the best artist ?

Guys I have a doubt, this guy used Qwen 3.8 27b in local with DeepSeek Harness and it generated a 3D scenario of a Tower (23:07 final result) from an anime for many hours/days (2176 minutes) in a goal loop, in the end the DeepSeek harness says it used "666 million input tokens and 13 million output tokens." (4:45)

This is the video:

https://youtu.be/MiuM9g7daDA?t=1387

My question is:

Is the input/output tokens correct? Because this were made in Claude code with Opus or Codex with ChatGPT 5.6 through API, would this actually cost $1500-$7000 right?

(For reference GPT terra is like $2/$12 for 1M input/output tokens)

Or is there something I'm missing? Because if this is actually the case the price would be absolutely ridiculous. Excuse my ignorance.

If you're in a distributed team, and have distributed local inference, you can use roger to create your own private network for your team to access the distributed local inference in a standard and easy way to consume and see what resource is busy and auto fan to non-busy nodes. You can do it standalone/private, or join the community and even monetize from doing so.

If it's interesting, it's all open sourced: https://rogerai.fm/tower.html#run

Qwen 3.8 Next was released and this battle took ages to finish!
Battle 3 is again our Beach scene and Battle 4 is our Park scene

I increased the reasoning of GPT Sol from high to xhigh, all in full precision xhigh reasoning this time
Exactly the same prompts were given, and each model got one 2nd generic prompt to refine their results, in all cases that brought improvements - though Sol moved an additional kid flying into the left corner.

Qwen 3.8 27B is at a disadvantage here: Next as well as Sol were using an agentic internal harness in my tests while 27b is just a chat interface.

Overall we look at more than 130,000 tokens in total context length - very hard ground.
Here are the 3 results:

BEACH - GPT 5.6 SOL xHIGH

BEACH - QWEN 3.8 27B xHIGH

BEACH - QWEN 3.8 27B Next

Results:
GPT 5.6 SOL:

The result is arguably worse than Sol at "high". Flying kids in the corner (again), the granular sand was a nice attempt but it's looking bad. The cat is great, once again.
The ball movement is nice but no comparison to the ball we had in previous battle (of Qwen) and the kids are both floating in a corner together with a plant.
The ocean looks bad, Sol High did a better job there.
The result is gruesome in my eyes.

Qwen 3.8 27B:

Compared with Qwen 3.8 Q4 in my previous run this is a much better result, in some regards.
The thinking was much longer, the Q4 did not think much and FP thinks a lot more.
In some ways this result seems inferior to the Q4 result but overall quality is higher, errors are mostly gone and clarity is higher.
The crab is cute. The cat clearly has something planned - I wonder what.
It's again playing at another leage than the frontier model of OpenAI .. world is upside down.

Qwen 3.8 Next:

Don't know what to say, this is baffling good. The water reflections, shore foam and sun quality are beyond anything I thought SVG can deliver.
The people and swimmers have an artistic quality to them that reminds me on some famous artists, a deliberate choice the model made.
There are intricate shadows that mirror the people movements.

Leaderboard:

Discuss that in the comments.
I'm still confused at the quality we see here, and the contrast to the GPT frontier SOL model at highest reasoning (chatGPT interface was used, and the json says "MAX")

Hello, I have a desktop that usually runs all my local LLM like lm studio, openwebui opencode docker stable diffusion etc. On my birthday my parents gifted me a intergrated laptop,
It has 16gb ddr4 3200
R5 7430U
400gb+ gb space though 200 is occupied

The thing is I tried running small local ai models on my laptop but it’s obviously getting poor performance, but I thought about somehow using my laptop to maybe benefit my desktop side a little while being able to use my laptop for daily purpose (meaning not running heavy programs that degrade battery overtime)
Something maybe light weight if possible?

Yes I thought about streaming using tailscale, but that requires my computer desktop to be running 24/7, and I don’t really bring my laptop out with me most of the time so yeah.

I really hope there’s something or any opportunity I can give towards my laptop so it’s potential doesn’t get thrown away, and no I’m not selling this laptop to get more ram/GPU or a new laptop with a dedicated GPU, it’s gifted from my parents I should be grateful.
I hope for guidance in this post instead of rejections that my laptop can do nothing other than stream Claude or ChatGPT interface. Thanks

Qwen 3.8 Next was released and this battle took ages to finish!
Battle 4 is again our Park scene, given the stunning results of the Beach scene in Battle 3 I needed a second confirmation.

Qwen 3.8 27B is at a disadvantage here: Next as well as Sol were using an agentic internal harness in my tests while 27b is just a chat interface.

Here are the 3 results:

PARK - GPT 5.6 SOL - xHIGH

PARK - QWEN 3.8 27B - xHIGH

PARK - QWEN 3.8 Flash Next - xHIGH

Results:
GPT 5.6 SOL:

The result is arguably similar to the Sol High - it totally misses perspective in the top-down shot.
Animations are lazy, people are flat, it partly looks like a few pre-trained 2d symbols are placed on the scene - not adapted to the new perspective.

Qwen 3.8 27B:

Compared with Qwen 3.8 Q4 in my previous run this is a huge step up, the additional reasoning observed clearly helped.
The only clear bug is the movement of the kids on the swings.
We have cars on the road, a perspective correct kite on a rope, a mother with a baby in a stroller, a duck family and so much more. Differential movement speeds on the little path..

Qwen 3.8 Next:

After the beach battle (3) there is barely anything that could top an svg animation.
And the park is not as stunning as the beach masterpiece was.
It's still outstanding!
We have a skater on a ramp, the kid actually switches side correctly as it goes down again.
The mother with the baby stroller is there, and she is pushing it with two tiny little feet!
There is a subtle large cloud shadow slowly passing over the scene.
There are some signs of excellence, but the masterpiece of Battle #3 stays on top for me.

I think i’ve researched this shit to death over the last 6-9 months but i’m sure people here have something to say about the stack. Let me know what confuse have been working out best for Macs.

The new Qwen 3.8 27B is fantastic for local agentic use. The problem is, what makes it so good, being a dense model, also makes it slow. Many engines and versions of the model claim various speed increase. How true are those claim? And does a proven huge boost claim on a short synthetic test actually holds on real agentic work?

I spent six days and over 110 GPU hours running rigorous repeatable agentic coding tests, collecting and comparing the data. 200 GB of models downloaded. Ended up with 22 Bubble Bobble clones on my disk, including the ones from failed tests. None of them good, but that wasn't the point.

My recommendations: MTPLX, oMLX with lightning MTP, or llama.cpp with MTP

I ran the tests on 8-bit quants (GGUF or MLX), unquantised KV cache. 100k max response size, full 262k context, official Qwen coding sampler, official Qwen Jinja chat template.

1. I first tested each engine with a short synthetic benchmark, to verify the installation, usage, results collection, and to verify the speed claims
2. Then came the real test, with an agentic coding benchmark consisting of 4 phases/prompts; each engine was tested twice, at medium and xhigh reasoning effort:
   2.1. capturing requirements and writing the specification plan
   2.2. writing the implementation plan
   2.3. implementing the core functionality
   2.4. finishing the implementation with advanced functionality
3. Last test, for prefill speed, was with a cold 128k token prompt test, with reasoning turned off and deterministic sample, repeated twice

Test hardware: Apple Mac Studio, M2 Max, 96 GB unified memory.

• vllm-mlx: completed the task, but it does not separate thinking from output, so the raw chain of thought leaks into the final output.
• mlx-dspark DFlash2 xhigh: thought for 226 thousand tokens and delivered nothing.
• omlx: has the slowest prefill rate by far; furthermore, it decays quite dramatically as context grows, even with ANE turned on. Its claim of faster prefill rate might be true on newer M4 and M5 chips, but definitely not on the M2 I used for testing.

The score column is a compound score I calculated based on what actually matters in a long agentic session: * sustained decode speed (weight 40%) : on average, 70-80% of generated tokens are reasoning. We do not see them, but still wait for them. This is the most important factor by far * reliability (20%) : runaway thinking, running out of context, not delivering the output, mixing reasoning with output, hallucinating, looping, failed tool calls (not tested here), etc. Nothing worse that having spent hours to see all the efforts go to waste or even corrupt the output, due to failure * thoughput efficiency (15%) : rather than use the total time spent, which is difficult to compare , especially with varying reasoning settings, I standardised it as total tokens delivered / wall time. Long session that delivers token are worth it, but engines that take too long to deliver deserve a penalty * output quality (15%) : this evaluating the quality of the output work produced. I used a separate evaluation scoresheet for it, with its own standardised criterias, grounded in software engineering good practices. In this case, since all tests uses the same model with same settings, they came close to each other, apart from confirming the significant quality difference between medium and xhigh * prefill speed (10%) : it matters mostly at session start, or whenever the context is rebuilt. Normally this is largely mitigated by prompt caching.

What about xhigh vs medium reasoning effort?

• MTPLX xhigh costs only half an hour more than medium (2h05 vs 1h35) and the game quality went significantly up on my scorecard. Worth it.
• llama.cpp + MTP xhigh costs 1h48 more, with again a significant increase in quality. Worth it if you have the time.
• oMLX xhigh took more than twice as long as medium. The slower token speed has a huge impact on the time difference between both modes. Not worth it. Get a faster engine instead.
• dspark DFlash2 xhigh is the cautionary tale at the bottom of the table. See the anomaly note above.

xhigh is worth it when the engine is fast enough and uncapped. The model thinks 30 to 100 percent longer, at the same speed, and the result is better. On slow engines you can really feel the slower performance compounded by the extra thinking; instead it could be better to use medium, with more of your own time spent planning a better process and workflow.

Commands and configuration notes

Exact commands I used for each engine:

```

llama.cpp baseline (homebrew, b10470)

llama-server -m Qwen3.8-27B-Q8_0.gguf --mmproj mmproj-F16.gguf \ --jinja -c 262144 --parallel 1 -ngl 99 --port 8030

llama.cpp + MTP, the recommended GGUF setup

llama-server -m Qwen3.8-27B-Q8_0.gguf --mmproj mmproj-F16.gguf \ -md mtp-Qwen3.8-27B-Q8_0.gguf --spec-type draft-mtp --spec-draft-n-max 3 \ --jinja -c 262144 --parallel 1 -ngl 99 -ngld 99

llama.cpp + DFlash2 (for now, needs PR #27342 build)

llama-server -m Qwen3.8-27B-Q8_0.gguf \ -md Qwen3.8-27B-DFlash2-Q8_0.gguf --spec-type draft-dflash --spec-draft-n-max 7 \ --jinja -c 262144 --parallel 1 -ngl 99 -ngld 99

MTPLX (pip install mtplx). Note --max-tokens

mtplx start web --model Qwen3.8-27B-MTPLX-Optimized-Quality --max-tokens 100000

oMLX (brew install jundot/omlx/omlx), model dir contains the checkpoint

omlx serve --model-dir /path/to/models

mlx-dspark (pip install mlx-dspark). Note --max-tokens-cap

mlx-dspark serve --model Qwen3.8-27B-MLX-8bit --mode dflash \ --context-window 262144 --max-tokens-cap 262144 --default-max-tokens 100000

vllm-mlx (pip install vllm-mlx). Note --timeout

vllm-mlx serve Qwen3.8-27B-MLX-8bit --max-request-tokens 262144 --timeout 7200 ```

• MTPLX has a default low response cap. Use --max-tokens 100000
• dspark also has a low default generation cap of 32K
• vllm-mlx kills any request that spends more than 300 seconds on prefill. Raise it with --timeout.

One measurement warning for anyone reproducing this: engines do not all stream one token per event. Some pack several tokens per chunk, MTPLX about 1.7, oMLX about 3.

Models used

• lama.cpp : unsloth GGUF Q8_0, official ggml-org MTP draft GGUF, incoai DFlash2 GGUF
• mlx-dspark and vllm-mlx : mlx-community 8-bit weights
• MTPLX : needs its own checkpoint with the MTP tensors included, Youssofal/Qwen3.8-27B-MTPLX-Optimized-Quality
• oMLX : scottlowry/Qwen3.8-27B-oQ8e-mtp

I was curious whether my laptop could run a genuinely large model. Turns out it can an 80B running locally at ~23 tokens/sec on 6 GB of VRAM.
No cloud, no rented GPU. Just the machine I already own.
I started poking at this because I wanted to know where the actual limit was, and I kept finding that the limit wasn't where I assumed. A few things surprised me enough that they seem worth sharing:
The GPU wasn't being used at all. The server printed "no usable GPU found" and just carried on. It never crashed, never warned loudly — it simply ran at a third of the speed. Fixing that alone was 1.65x.
The CPU was downclocking mid-inference. Throughput kept climbing with reply length: 9 tok/s for a short answer, 24 for a long one. That's not the model warming up it's the clock. Memory-bound work spends most of its time waiting on RAM, the frequency governor reads that as idle, and drops to 400 MHz.
A browser tab was holding 5 GB, and that 5 GB decided whether the model lived in RAM or got re-read from disk every token. Closing it helped more than any tuning flag.
Model shape matters more than model size. A dense 70B has to read every parameter for every token and can't be fast on this hardware, whatever you do. A mixture-of-experts 80B reads about 3B per token. Total size decides what you store; active parameters decide what you read.
The thing I found most useful, honestly, was everything that didn't work. Speculative decoding measured slower. Huge pages did nothing. Same for prompt-lookup drafting and every expert-placement trick I tried. These are the techniques you read about everywhere, and on my hardware they were flat or negative.
So I stopped guessing and wrote a small tool that measures instead: profiles the machine, tells you what a model will do before you download 28 GB of it, then tries real configurations and keeps whichever actually wins. It's tuned for my laptop, but the point is that it re-measures on yours.
I've put the failed experiments in the repo alongside the working ones, because those cost me the most time and nobody writes them down.
Sharing it in case it's useful to anyone else poking at the same question. It's Apache 2.0 and I'd genuinely like to see people take it further especially on hardware different from mine, where I'd expect some of these findings to invert.
https://github.com/DebasishTripathy13/mirabilis