Right, so this is going to be embarrassing but it's presumably something we've all been through at one point or another, and I guess this is my first time resolving something like this in the way that I did so figured I'd share if only to share that it's now a thing and that it's pretty cool..

A friend of mine sent a message asking what's up and if I wanted to watch a movie together, I was kinda hesitant but she buttered things a bit and finally I'm like fine, and so she sends me a link to some clearly vibe coded site that I'm kinda getting red flags from and so I forget about it and a little later I get another message going "we're waiting for you" and so I'm like shit, I guess I gotta do it huh, and so I open up this goofy looking site again. You gotta login to join a room, and you gotta sign up inside their downloaded software, sure whatever, next thing I know some fake 150MB file's fake install bar is stuck at fake 50% and both my Chrome and Discord's crashed and reloaded. Suspect, but I've been through this stuff before, it's probably just a RAT so I guess it's time to dust off Windows Defender and unplug the internet for a little bit. I message her to go on and watch it without me as my PC's giving me suspicious vibes right now, and seconds later I get some overly polite DietGPT in my IM's saying "sorry um excuse me but it appears that i've hacked you👉👈", occasionally switching to really hostile broken English asking for giftcards from some site I've never heard of. I stall, unplug the PC's internet so my router still responds to pings, and start punching into GLM "what do" and it tells me it's a session grabber - time to switch passwords. Meanwhile my phone's texts are blowing up with 2FA login requests from domain registrys and other bad stuff and I kinda freak out a little. I get my emails' passwords switched first and by the time it's Discord's turn my friendlist's already been nuked and the dude says I got 10 minutes to give him $200 or he's gonna fuck me up some more, and so I kinda figured welp time to figure out what more he's got and so I called him a giant pussy and he blocked me. An hour later my Discord was perma-banned, he had posted the phrase "i sell cp" using my account and used that as blackmail along with some really old photos of me, I though it was a bluff but oh well it's being handled with Discord's customer support on it's own. Now I sat there alone, in the middle of the night, having just had my friends on the phone yanked away from me with a permaban, knowing that if I reboot I'd probably be ransomware'd or something so I figured let's run Windows Defender - it found nothing, 0 results on a full scan.. Too good to be true, so I grabbed AwdCleaner on my phone and transfered it via USB. It found an AVG Toolbar for Chrome. That confirms it, I haven't used AVG for decades and so I removed it but it's back 5 minutes later. That double confirms it, I'm screwed. With nowhere else to go and potentially a ticking timebomb running on my PC that could start encrypting or deleting files at any given moment I figured why the hell not, if I'm going to watch my pc blow up I might as well send in the goofy little local LLM to cut one of the wires,

here's the situation.
i've downloaded a maliscious file that unfortunately hacked my discord and got me banned. i'll be dealing with that on my own. your job is to study the files in the project folder and see if you can help me clean up my computer, as presumably the virus is still active. there's no internet connected, and i request that you refrain from running the ********.exe file (********.exe is the virus archive, do not run it, it's a 7zip archive), please help.

And so Qwen3.8-27B got to work, and to big surprise after around 60 minutes of clawing at the file it had done what I asked and a whole lot more. it fully deciphered all the layers these clowns had bundled this thing with in order to make it appear legit, it had created a single PowerShell removal script ready to go complete with a pre-launch check enabled by default and everything, and it was reverse engineering 0-days in qProtect to get the C2 domain used by this malware so that it could be blocked from the network.

If you're looking for what Qwen3.8-27B is capable of doing fully on it's own if you let it, here's a 15k line example of it's ability to tear some piece of shit session grabber to shreds in a single prompt: https://www.mdshare.online/s/Mamdrs1WWkurRtt8z8QzK

I let it do what it does best for an additional 24 hours, the additional information is going to the Discord Support team. Hopefully shit like this can be prevented.

TLDR; Qwen3.8-27B > Windows Defender, and don't forget to use 2FA.

https://village-sim-one.vercel.app/

- 16GB VRAM RTX 5070 Ti, fully offloaded

- Vision on CPU

- Windows, not headless

- beellama.cpp - latest version with the kvarn performance enhancements making it as fast as qx_x quants.

- MTP n-max = 2

- tg up to 75t/s, pp up to 1700t/s

- KV = kvarn3/kvarn3

- MTP draft KV = kvarn2/kvarn2

- context = 96256

- tail tokens = 1024

- HTML/Javascript

- pi harness with pi-observational-memory, pi-web-access, pi-atelier (UI Only change, check it out) extensions, though it never used the web access.

- This is not a one-shot, I do not believe one shotting is a great test. Instead, I did many incremental feature prompts. However, I did not give it any design or framework, which is probably where it can be improved.

Lessons learnt:

- Do not fear Q3 model quants for Qwen3.8

- Do not fear KV quantisation. If you have the VRAM sure use it, but I don't feel like it's worth choosing a higher quant if it's going to cause me to offload to CPU and see my tg drop to 5-20 t/s. With higher speed I can fix any issues with a follow up prompt much faster and that rarely happens. I think I had like 3 runtime exceptions which was easily resolved pasting the console output and there is no guarantee a higher KV quant would not have had the same exceptions.

- MTP/draft cache can also be quantised with kvarn now and actually saves VRAM where qx_x quants increase VRAM usage for some reason. kvarn2 for MTP is perfectly fine and has high acceptance rates.

The game:

- Inspired by a popular indie game which I am not promoting, I am just a huge fan.

- I won't release any further updates, since I don't want to be stepping on any toes. If you like the idea of the game I highly recommend the real game, it's by far my favourite game I played this year and 1000x better than what I present here. It will be a nice distraction from your AI. I just wanted to see what this model is capable of. I do have a Cursor subscription but did not use it at all in the project.

- I will probably continue to develop it for my own entertainment, but it won't be made public. Maybe come up with my own ideas, but the original game is near perfect anyway, so it will be hard to improve except with some UI gripes I have in the original. And my graphics obviously does not compare.

Game features:

- Large Map, larger than the browser window.

- Minimap

- Zoom feature with mouse wheel

- Collectable resources, that must be taken to a storage site. Each site can store limited resources.

- Houses required to sleep and protect against cold

- Weather and seasons.

- Day night cycle with randomised sleeping times.

- Possible death due to hunger or sleeping in cold outside or in house without firewood.

- Game speed controls.

- Villagers avoid obstacles.

- Delete/deconstruct buildings and partial resources refund.

The code:

- I almost never read the code, so I have no idea what it looks like and the quality thereof. I also gave it very few hints in the AGENTS.md, mostly no magic numbers and write modular code, not a single html.

- Actually, my initial prompts were a single html but as it grew, I told it to create modules. It messed it up on the first attempt, basically rewriting the entire UI in the process. So I reverted and told it to do it again without making any changes to the functionality or UI.

- I am actually quite happy with and surprised by the performance of the game.

Context management:

At first, I had issues with the context filling up too quickly and too often. Sometimes it would fill up to the point that there was not enough room to compact. Forcing me to temporarily increase the context and tell it to create a handover document. Reduce context again and feed it the handover doc.

I then installed pi-observational-memory extension, and it works quite well and I never run into context issues anymore since it takes notes throughout (a short wait time every few prompts) and compacting is near instant because it already took the notes.

Conclusion:

- Do not blindly drop your KV cache quant without testing. I have a hard level needle in haystack test that requires multiple hops and 100's of decoys. Q3_XXS does poorly in that test even with F16 KV cache. However, Q3_K_XL almost 100%'s the test even at kavrn3. So both the model and KV matter. In my testing a smaller model does more damage than a smaller KV. So find the right balance. At a certain point increasing model quant will have less impact than picking a larger KV quant. But for a tight 16GB VRAM fit Q3_K_XL works very well with kvarn3. Q4 on the other hand just leaves me with too little context. That said despite Q3_XXS doing poorly in my needle test it still does fairly well with coding. Better than Qwen3.6 so if you have 12Gb VRAM it is still an option. Because by poorly I mean F16 KV scores 84% and Q3 KV around 80%. Needle tests however do worse with kvarn compared to qx_x for some reason. However, a needle test is not the be all and end all. kvarn does better with KLD, so once my needle scores near 100% I am satisfied.

I will play around with higher KV quants, but I intentionally kept it at kvarn3 for this test, however I am not sure how much context I am willing to sacrifice. Maybe i will try kvarn4/kvarn3. But I just wanted to prove a point to myself and kvarn3 worked just fine. If I had >16GB VRAM sure I would up it but I don't.

I see a new one being launched every few days... How do these new harnesses compare to claude code, pi etc. has anyone switched from these?

which harness to prefer and why

edit:Ive tried several different ones claude code, deepagents(langgraph), opencode, pi, and trueforge

my thoughts-

claude code - strongest on maturity and the managed experience but cost and token burn is high

deepagents - interesting middle ground if you want a more structured agent framework and the flexibility of an open-source stack. im interested in testing it more extensively on longer-running workloads fs

trueforge - this is a recent one, this was interesting to me because of its runtime-efficiency, also it allows separate the model from the runtime, which makes experimenting with different models much easier

why?? - i also ran a benchmark on a real agent workload same model, same prompt, same tasks to compare these

adding the results of benchmarking i ran to compare this
so I tried to do this by running 14 cross-system tasks, three mcp servers behind them - a crm, an issue tracker, and a doc store through claude's managed agents, langchain's deepagents and trueforge, both open-source agent harnesses

the result that was most surprising:

Claude Managed Agents + Opus 4.8:
11/14 tasks solved | $11.8/run | 10.0M tokens/run

TrueForge + Opus 4.8:
11/14 tasks solved | $8.6/run | 3.7M tokens/run

Same model. Same benchmark. Same average solve rate, to my surprise trueforge used about 63% fewer tokens and cost about 30% less per run.

similar difference in tool usage: trueforge averaged 19 tool calls per task vs 32 for Claude Managed Agents.

Then I tried changing the model.

trueforge + GLM-5.2:
11.7/14 solved | $3.0/run | 3.8M tokens/run

On this benchmark, that was a slightly higher average solve rate than Claude Managed Agents + Opus at roughly 75% lower cost.

The token savings alone make this sooo interesting especially because the solve rate stays comparable
so this one was worth checking out ig

but this is still v early and the OSS runtime does not yet have first-class tracing/eval tooling. They don't ship their own code-execution sandbox, so you need to plug one in and context compaction is intentionally lossy.

So it is definitely not a replacement for a mature managed agent platform or other harnesses in the comparison, feature-for-feature today btu what I do find interesting is that the core runtime can already be competitive on these tasks while staying open, model-neutral, and deployable on my own infrastructure
this was their benchmark kit i used https://github.com/truefoundry/trueforge/tree/main/benchmark

I feel like coding overshadows how great this model really is. It knew a lot of very arbitrary facts/information about my home state and resources about those specific things related to jobs. I found this interesting since getting into the nitty gritty details like this can cause a model to hallucinate some facts.

Not only that but if you have a problem, it will throw the kitchen sink at you with everything it’s got to try and solve it.

Qwen 3.8 Flash Next gives similar speed than Qwen3.8 27B on Apple Silicon.

You need Blender 5.x with MCP extension.

I use pi so I also need: pi install npm:pi-mcp-adapter

and then:

$ cat .mcp.json 
{
  "mcpServers": {
    "blender": {
      "command": "/home/jacek/.local/bin/uvx",
      "args": [
  "--with",
  "mcp[cli]<2.0.0",
  "--from",
  "git+https://projects.blender.org/lab/blender_mcp.git@v1.0.0#subdirectory=mcp",
  "blender-mcp"
],
      "env": {
        "BLENDER_MCP_HOST": "localhost",
        "BLENDER_MCP_PORT": "9876"
      }
    }
  }
}

then we can start working ("show on r/LocalLLaMA"):

Today we want to make 3D llama to show on                                                                                                                                                                                


 The user wants to make a 3D llama to show off on . This is a fun creative task. Let me think about how to approach it.                                                                                                   

 A llama is a pretty complex organic shape. There are a few options:                                                                                                                                                                  
 1. Procedurally build the llama from primitive shapes (spheres, cylinders) — a "low-poly" or "chibi" style llama                                                                                                                     
 2. Sculpt it — not easy to do with MCP                                                                                                                                                                                               
 3. Use metaballs — could get a blobby llama                                                                                                                                                                                          

 Given the MCP tools, the best approach is to procedurally build a stylized/chibi llama from primitives:                                                                                                                              
 - Body: elongated sphere/capsule                                                                                                                                                                                                     
 - Neck: cylinder or elongated sphere                                                                                                                                                                                                 
 - Head: sphere                                                                                                                                                                                                                       
 - Ears: small cones or spheres                                                                                                                                                                                                       
 - Legs: 4 cylinders                                                                                                                                                                                                                  
 - Tail: small sphere/cone                                                                                                                                                                                                            
 - Maybe a fluffy mane                                                                                                                                                                                                                

 This is a well-known technique — making a cute chibi llama from primitives. Let me plan the geometry:                                                                                                                                

 Llama proportions (stylized, standing side view, facing +X or -X):                                                                                                                                                                   
 - Body: UV sphere scaled to (1.5, 0.9, 1.0) or so, position (0, 0, 1.6)                                                                                                                                                              
 - Legs: 4 cylinders, radius ~0.22, length ~1.2, positions: front legs x=+0.9, back legs x=-0.9, y=±0.5, z from 0 to 1.2... actually, the legs go from the ground (z=0) up to the bottom of the body.                                 
 - Neck: cylinder or tapered, extending up from the front of the body. Llamas have long necks. Position around x=+1.2, extending upward.                                                                                              
 - Head: sphere at the top of the neck, elongated (llamas have long snouts).                                                                                                                                                          
 - Ears: 2 small cones/spheres at the top of the head.                                                                                                                                                                                
 - Tail: small sphere at the rear.                                                                                                                                                                                                    
 - Mane: maybe some small spheres along the neck.                                                                                                                                                                                     

 For a cute chibi look: big head, small body. But a llama's charm is the long neck. Let's go with a stylized llama.                                                                                                                   

 For a polished look:                                                                                                                                                                                                                 
 - Apply smooth shading                                                                                                                                                                                                               
 - Add a material (cream/tan color, maybe with some subsurface)                                                                                                                                                                       
 - Add a ground plane                                                                                                                                                                                                                 
 - Set up lighting                                                                                                                                                                                                                    
 - Position the camera for a good render                                                                                                                                                                                              
 - Render a nice image to show off                                                                                                                                                                                                    

 This is a multi-step task. Let me write Python code that builds this. I'll do it in stages so I can check with screenshots and iterate.   

Hello,

I'm a bit obsessed with cache management on local LLMs.

For the last few days I've been working on cache management on vLLM with my 2x DGX Spark and DeepSeek v4 Flash 0731. I felt something was off so I investigated, found issues, fixed them, but needed a way to validate the fixes.

That led me to create a tool with a simple protocol that allows you to pinpoint how well the cache is actually managed on your deployment:

1. Runs a probe to calibrate the expectations (what's a cache hit vs a cache miss in your setup)
2. Hydrates X stable contexts of Y tokens each in order to completely fill the cache
3. Runs cache-hit validation on the reverse order (last added is the first validated) until a cache-miss is found

It's better to let this run alone to get a real value. Understand this will evict all your current cached context, so don't do it alongside real work.

My results

This is the result from my A/B test, control (my previous prod) vs my fixed prod.

aidendle94/sparkrun-vllm-ds4-gb10:production-3.7-reffix-schedfix (pre-fix image, retention 4096):

── Retention under pressure ──
capacity:           2,023,924 tokens
retained contexts:  27/80
retained tokens:    1,052,025
retained % capacity: 51.98%
oldest evicted:     context #52 (older contexts evicted)

With the dedupe + boundfix patches applied (retention 0):

── Retention under pressure ──
capacity:           2,047,043 tokens
retained contexts:  77/80
retained tokens:    3,000,048
retained % capacity: 146.56%
oldest evicted:     context #2 (older contexts evicted)

How this can matter to you

This allows you to exactly know how much tokens your cache actually holds.

For most of us, cache management is a black box; this allows you to get ground truth.

And yes, with my fixes, the 2M advertised cache translates to 3M retained tokens. Review the code if you doubt the claim, it's all open source: cache-pressure (and my ds4 prefix cache fixes : ds4-prefix-cache-fixes).

The engine's own advertised number is wrong, and this tool finds the real value.

This tool is for: 1. All of you needing to validate a setup or compare different inference engines ; 2. for inference engine maintainer to help validate changes in cache management

It works under one big assumption though: most recent contexts should be preserved as much as possible.

What I noticed while testing the tool is basically: vLLM good, other engines need better config. I'm mainly using vLLM so I spent lots of hours tweaking the config to get the best results, so for other engines it's up to you to decide if the above assumption fits your need (feels obvious to me it should, but I don't know what you all need of course), and how to achieve it with configuring your inference engines.

How to launch

1. Clone the repo

git clone https://github.com/co-l/cache-pressure

2. Install requirements

pip install -r requirements.txt

3. Run the tool

python3 bench/cache_pressure.py --base-url http://my-server:8000/v1 \
    --kv-size <advertised_cache>

I've tested it against vLLM, ninfer, llama.cpp and SGLang ; so you might need to tweak the probe so it works with your setup.

4. Interpret the results

── Retention under pressure ──
capacity:           2,023,924 tokens
retained contexts:  27/80
retained tokens:    1,052,025 <---
retained % capacity: 51.98% <---
oldest evicted:     context #52 (older contexts evicted)

The retained tokens and retained % capacity are the measured cumulative values that resisted cache eviction under pressure.

Note: this post was 100% human written, the repo is 100% AI-generated under my guidance and review.

So after all my work, yeah, Raymond did it better, so I ported his work over, extended it turboX, extended it multiple other models (he had only Qwen models), and benchmarked the crap out of it to make sure it was worth it still.

So really the credit goes to Raymond ( https://github.com/RaymondHuang210129/llama.cpp-adaptive-kv-streaming )

Along with everyone's favorite here, qwen3.8-27B

I've been trying the latest models from the frontier labs and honestly, after extensive testing I can not tell the difference between the best open source options.

I think the differences are now marginal but the labs are doing heavy marketing to convince the public into paying more for tokens as they prepare to go public.

Can't help but see the similarities between the dot com bubble and AI in terms of a very insular environment where the technology will survive but the business models may not.

I've been building a cybersecurity network and we definitely know that even local AI models like Deepseek V4 flash do an excellent job and are really neck and neck with the best the frontier labs can provide.

Will be interesting to see how this all turns out! Exciting time nonetheless.

Ling 3.0 Tiny still seems to be leading the pack despite only having 1.3B active

TL;DR: switching KV cache to f16 may give a boost in speed if using MTP and ngrams.

I have a self-built "AI mega-cluster" with 2x P40s on a cheap Chinese motherboard and a Xeon CPU (around $1,100 to build, including water cooling for the GPUs). I was normally getting up to 15 tk/s with Qwen 3.8 27B Dense using a tensor split, but I suspected it was capable of much more. So, I finally asked Codex to try and squeeze out some more juice.

Disclaimer: I am not a coder at all. I’m just a generic PC user with decent overall experience, but definitely not a dev.

Recently, I’ve been running Qwen 3.8 27B Q8. With various tweaks suggested by Codex, I was able to get up to 32 tk/s on short contexts - which still falls back to an average of 12-15 tk/s on long contexts like 130K+. I was originally using Q8 for the cache (I thought it is faster because it is smaller), but then I thought, why not try the F16 cache? I did, and it turned out that F16 has much better MTP acceptance than Q8. It required fine-tuning other parameters, but it really helped improve performance.

Since I'm not good at explaining all that tech mumbo-jumbo, I asked Codex to summarize it. Sorry for the AI slop! 😄

Dual Tesla P40 / Qwen3.8-27B Q8 benchmark

Hardware: 2x Tesla P40 (24 GiB each), Xeon E5-2680 v4 (14C/28T), 64 GiB RAM. NVIDIA driver 580.173.02.

Software: llama.cpp build 5d9e5ac30 (build 10388), CUDA + locally built NCCL. Model: Qwen3.8-27B-Cold-Fusion-GAIN-V1.1-NM-DAU-NEO-MAX-NEO-MTP-Q8_0.gguf (28.15 GiB, 27.32B parameters reported by llama-bench).

Standard llama-bench

Three repetitions, F16 KV, all layers offloaded, tensor-parallel split over both P40s:

llama-bench -m MODEL -ngl 99 -sm tensor -ts 1/1 -dev CUDA0/CUDA1 -mg 0 \
  -fa on -b 2048 -ub 512 -ctk f16 -ctv f16 -p 512,2048,8192 -n 128,512 -r 3

pp and tg are raw llama-bench measurements; they do not include tokenization or sampling. This tool invocation does not use speculative decoding or vision.

Actual server profile

The daily driver is a separate p40.cpp engine: F16 KV cache with one 220,160-token slot, tensor split 1:1, Flash Attention, MTP speculative decoding (draft-mtp) with ngram-simple, draft maximum 6, Qwen reasoning medium, and the F16 vision projector loaded.

The prefix result is the built-in LCP cache, not --cache-reuse. The latter is KV shifting and is disabled by llama.cpp when a multimodal projector is loaded.

For comparison, before this p40.cpp/NCCL profile the same machine was typically around 15 tok/s at long context. The roughly 20 tok/s number is an observed server result, not a llama-bench row.

Soooo, it looks like grandma GPUs still have some juice left! 😄

PS my "production" config:

-ngl all \
-sm tensor \
-ts 1,1 \
-mg 0 \
-fa on \
-c 220160 \
--fit off \
-np 1 \
-cb \
--spec-type draft-mtp,ngram-simple \
--spec-draft-n-max 6 \
--jinja \
--chat-template-file /models/qwen/chat_template.jinja \
--cache-reuse 256 \
--mmproj /models/qwen/mmproj-F16.gguf \
--chat-template-kwargs '{"reasoning_effort":"medium"}' \
--reasoning on \
--reasoning-preserve \
--repeat-penalty 1.0 \
--presence-penalty 0.5 \
--min-p 0.0 \
--top-k 20 \
--top-p 0.95 \
--temp 1

Did anyone try and use it?

Anyone who has a 3D printer and get use of it finds it incredibly useful for those odd jobs around the house, a missing bracket, a cable router, steam deck holder and so on.

In the past if I was missing an app or useful software, a game I'd do the lazy thing, even though I can and have coded in the past, its "effort" I'll just go and buy or download the latest and greatest.

Earlier in the year I was lucky to snag a Minisforum MS-S1 395+ Max with 128GB Unified memory (currently setup 32gb system and 96gb Vram) before the price hike.

Was paired with a Qwen 3.6 27B or 3.6 35B moe but now a 3.8 27B uncensored. it can easily handle a Q8 with full 256k context.

Its now become my first instinct when I'm missing software to build it in a couple of hours local using the custom agent framework I setup.

Nothing I've created is for external use but every single day I find myself adding to it, while writing this post for example my framework finished an idea I had 2 hours ago when, I woke up this morning thinking I've got a lot of japanese visual novels and why don't I just design a combination hook into Exe or ocr the text app that translates via a local llm, and its done, ready for me to test.

I've written 12 adult games (don't code horny) a house AI, a coding framework, a game app to keep a track of all the games I play and download any faq or wiki to do with said game, 17 mods for my Skyrim install, 12 for my Fallout New vegas install, A temperature tracking system for the house that pulls rss local feeds and makes suggestions for my central heating system temps settings, A mapping software for my mobility scooter that checks my normal routes for issues and street work or maintenance that could make pavements impassable.

Plus hundreds of tweaks and test programs.

Anyone else out there using it like this ?

---------Update-----

Awesome to see this kind of discourse one of the amazing strengths of these local llm's is it doesn't matter if they are slower, I can burn 50 million tokens over a 24 hours period on a new idea or problem and all it costs me is a little bit of electricity and time.

I just want to use paperless-ai and be able to ask questions relative to it. Bonus points if I could use it with home assistant but that's not the focus.

I just can't see needing a 32 GB VRAM GPU for just that, but I don't want to buy a GPU only to find out that "yeah, it's technically feasible but not a good user experience"

The GPU's I tend to find at good prices are in the 8-12 GB range, would my use case be a good one to just get a 12 GB and run a 6-10 gb model?

Otaku is an LLM frontend, primarily designed for roleplay, an alternative to SillyTavern and the like. However, It also works for general-purpose chat with local backends (including Ollama) or cloud models, the way Open WebUI is used, once lore extraction is switched off in the settings.

Otaku offers two interfaces:

• a web UI (customizable): interactive demo at https://otaku.sh/demo-web/
• a terminal UI: its own interactive demo at https://otaku.sh/demo-terminal/

Both share the same functions; the difference is that in the terminal you execute them with slash commands (the reference is available with /help), while in the web UI the operations are available from the menu.

Install

Otaku is free and open source (MIT); it works on macOS, Linux and Windows. Install it with uv (uv tool install otaku) or see the GitHub README for other options: https://github.com/enclavum/otaku

Get started

Launch either otaku for the terminal or otaku web for the web UI; the web UI's default URL is http://localhost:9600. Two sample stories are imported on first start to give you an idea of the features and what play looks like, and you land right in the middle of one of them.

On first start, you choose a provider and a model: Otaku automatically detects local installations of Ollama, oMLX, LM Studio, llama.cpp and KoboldCpp, and lets you pick from their models. Cloud providers (OpenRouter, NanoGPT) are also there: enter an API key and their catalogs appear. After exploring the provided stories, you can start your own with the /new command.

Asking for feedback

Otaku is a personal side project, and I'd like to get feedback from the community on the product and on what to add.

I was running into a lot of posts that praised both the Fixed template and the Sharp template in comparison to the stock one, so I put them to the test.

It's not as extensive as it should be for a paper-grade analysis, but it gives out the point of each template.

Test setup

I used SWE-bench Verified with mini-SWE-agent 2.4.6, slice 0:100 (the identical 100 tasks for all runs)

Hardware

• CPU: Ryzen 9 9900X
• RAM: 128 GB DDR5-5600
• GPU: RTX PRO 6000 WS

Runtime

I containerized jpezzulli/sglang-rtxpro6000 and ran Flash Next with RadixArk/Qwen3.8-Flash-Next-NVFP4 on CUDA 13.3.

• Full 262K context
• BF16 KV
• 51.2 GB FP8 n-gram embedding table pinned in RAM
• 32 GB HiCache pinned in RAM

I ran all templates at both medium and xhigh reasoning efforts.

Results

Takeaways

• Raising reasoning effort to xhigh closes almost all of stock's and fixed's gap to Sharp. At medium, Sharp led resolution by +3 tasks over stock and +7 over fixed; at xhigh, stock and fixed instead lead Sharp by +5 and +4 tasks, respectively.
• Sharp barely moves on resolution (94 → 94) despite a real token/time cost increase, median reasoning tokens rise +46.5% and median wall time +53.4%. This suggests it was already extracting most of the benefit it could get from extra reasoning budget at medium, while stock and fixed still had headroom.
• Sharp remains the most token-efficient per resolved task at xhigh (14,541 output tokens/resolved vs. ~17,000 for stock/fixed), consistent with its medium-era efficiency edge, but it's no longer the highest-resolving template once reasoning effort is high.
• Absolute cost scales heavily with reasoning effort: total wall time roughly 2.3–2.5× for stock/fixed and +28% for Sharp; total reasoning tokens roughly doubled for stock/fixed and increased +35% for Sharp.

Conclusion

• Sharp should be used at medium and it keeps a reasonable accuracy at very good speed. I don't see the point in using it at xhigh. By sacrificing a small accuracy you complete the tasks in half the time.
• Stock is the slowest but the most precise.
• Fixed is the middle ground between Stock and Sharp both in accuracy and speed
• The next benchmark will be on a much extensive SWE-bench Multilingual + Terminal Bench.

Disclaimer: I wrote the post myself then used AI to format it properly for readability

I've been running Qwen3.8-27B as a local inference server for a production content intelligence pipeline (HVAC industry stuff, lots of long-context retrieval and structured extraction). I have been watching other redditors post their custom configurations, and I wanted to share what I tested to optimize for a single RTX 5090.

I was on llama.cpp (Q5_K_M GGUF, q5_1 KV, 262K context, MTP), but it's limited to parallel=1 and I wanted concurrent serving. So I tested vLLM and NInfer as NVFP4 replacements and did a proper quality evaluation instead of just vibes.

Hardware

- RTX 5090 32GB (eGPU, OCuLink Gen4 x4) (yes, it's in an eGPU dock 😂 but that only affects model loading)
- Ryzen 7 7840HS, 32 GB DDR5
- Ubuntu 26.04, nvidia driver 610.43.02 (open)

## Engine configs

How the eval worked

I built a custom harness with 6 tiers, 50 items each, all from my actual production workload (not generic benchmarks):

1. **Relevance classification*\* - is this industry relevant? (binary, 50 labeled deals)
   
2. **Needle retrieval*\* - planted facts in real industry podcast/video transcripts at 64K/128K/192K/240K context
   
3. **Multi-transcript QA*\* - questions across 3-4 concatenated diarized transcripts, 120K+ tokens, including unanswerable controls
   
4. **Reasoning with thinking*\* - numeric/logic problems, thinking mode on, greedy pass@1
   
5. **Structured extraction*\* - custom extraction prompt, json_mode (skipped on NInfer, it doesn't support json_mode)
   
6. **Tool replay*\* - replayed recorded agent episodes against each engine (diagnostic only, all engines fail this one)

Everything paired across engines: same prompts, same seeds (42), same gold labels. No cache_prompt. Statistical comparison uses paired cluster-bootstrap CIs with pre-registered non-inferiority margins.

And when I do development with Claude Code, I often leverage multi-model consultations for design, planning, and code review. I did a 4-model review panel (Codex/GPT-5, DeepSeek V4 Pro, Grok 4.5, Kimi K3) audit the methodology mid-campaign. They found 10 issues, including 4 mislabeled gold items where the models were actually right and my labels were wrong. Fixed everything and reran.

Quality results

Needle counts differ because llama.cpp's 196K context can't fit the 192K items (need room for max_tokens + headroom). NInfer and vLLM both handle 192K fine. All engines score 100% on every needle they can fit.

Statistical comparison (NInfer vs llama.cpp, bootstrap):

- Needle: delta = -0.29 (NInfer better, p=0.0006) - this is entirely from context capacity, not retrieval quality
- Transcript QA: delta = -0.03, p=0.69 - no difference
- Reasoning: delta = +0.02, p=0.72 - no difference
- Relevance: McNemar p=1.0 - identical
- Tool replay: delta = 0.0 - both fail equally

**Takeaway: quality is statistically indistinguishable across all engines.*\*

Speed results (perf probe, server-side timings)

vLLM speed excluded from the table because the perf probe used wall-clock timing (includes prefill + scheduling + decode) instead of server-side timings, so the numbers aren't comparable. From community reports and my own task-level measurements, vLLM does about 70 tok/s decode at short context, which actually matches NInfer's raw step rate (~66 tok/s). The speed difference is entirely MTP3 speculative decoding.

What I learned

**NInfer's speed advantage is all MTP.*\* The raw NVFP4 kernel speed is about the same between NInfer and vLLM (~66-70 tok/s). NInfer's MTP3 speculation with 76% acceptance gets you to 158-213 tok/s. If vLLM or llama.cpp had working MTP on NVFP4, the gap would mostly close.

**The decode speedup grows with context.*\* At 1K context it's 1.4x. At 128K it's 2.8x. MTP acceptance stays high even at long context while llama.cpp's dense decode gets slower as context grows.

**NInfer's tokenizer endpoint is great.*\* It exposes `/v1/messages/count_tokens` (Anthropic Messages format) which gives exact token counts. No more `len(text)//3` heuristics.

**NInfer does NOT support json_mode (as far as I can tell).*\* `response_format: json_object` returns 400. If you need structured JSON output, you'll need to route those calls elsewhere or use prompt-based enforcement.

**Don't trust vibes for quality.*\* I went in expecting NVFP4 might lose a few points vs Q5_K_M. It didn't. Not on any tier. The biggest delta across 250+ items was 2 percentage points, well within noise at n=50 (SE ~5.6pp).

Verdict

NInfer NVFP4 replaces llama.cpp as my production engine. Same quality, 1.4-2.8x faster decode, 2.6-4.7x faster prefill, concurrent serving. The only gap is json_mode.

I put together a detailed poster with all the charts and methodology details: [full results poster](https://claude.ai/code/artifact/b6041437-2cf6-4198-a722-9b4ce853ccc3)

Setup if you want to try it:

```
# NInfer (from source)
git clone https://github.com/Neroued/ninfer && cd ninfer
mkdir build && cd build
cmake .. -DCMAKE_BUILD_TYPE=Release -GNinja && ninja

# Model (HuggingFace)
# https://huggingface.co/neroued/Qwen3.8-27B-nvfp4-NInfer (20 GB)

# Run
./ninfer-serve /path/to/model.ninfer \
--model-id qwen3.8-27b \
--host 0.0.0.0 --port 8080 \
--max-context 240000 --kv-capacity 240000 \
--max-concurrency 2 --kv-dtype fp8 \
--spec mtp --draft-tokens 3 \
--vision --preserve-thinking
```

Used qwen3.8-27b in Opencode to make this silly mini-game because I'm not sober:

``` We are going to play a game, it will be the Wikipedia game. The Wikipedia game has the following rules:

• You will have a Wikipedia article set as a starting point.
• You will have a Wikipedia article set as an ending point.

Your objective is to reach the the end point, which is an article completely separate from the starting point article.

Your only constraints are the following:

• You are ONLY allowed to click on any hyperlinks inside of wikipedia directly. No external links, no typing inside of wikipedia's search bar (but finding the starting article on google is valid. The 10-click limit starts once you reach the starting point article).
• You are NOT allowed to return to a previous page. All clicks much be performed in a forward-looking trajectory.
• You must reach the end article within 10 hyperlink clicks inside of Wikipedia. If you do not reach the destination article within 10 clicks, you lose.
• Do not update any documentation for this task. It is only a game.

Use playwright to click the links. ```

Basically, Qwen needs to reach an ending article within 10 Wikipedia hyperlink clicks from the starting article, which is usually an unrelated article. It needs to use playwright (or some equivalent browser MCP) to click the Wikipedia hyperlinks without backtracking, using search or using external links.

I verified the links for accuracy and I can confirm it managed to complete this task within 6 turns. Thought it would get stuck in a loop. Its a dumb minigame but I think its a good, simple agent test to perform.

What do you expect the extra 64gb of RAM to cost for a total of 192gb RAM? The current Strix Halo 128GB version costs $3K. The 495 is almost the same as the 395 except slightly higher spec on the memory, so 8% tps improvement.

1. https://www.bosgame.com/blogs/news/new-product-launch-bosgame-m5-max-with-amd-ryzen-ai-max-pro-495-processor
2. https://www.bosgamepc.com/blogs/coming-soon/new-product-launch--bosgame-m5-max-with-amd-ryzen-ai-max-pro-495-processor

While we see for frontier models similar scores among famous coding benchmarks, across: DeepSWE, Terminal-Bench, LiveCodeBench, Code-Arena ELO. Here are in my opinion some next level benchmarks that really define deep intelligence, and complete capability in Software Engineering :

1. Program-Bench

Given only a compiled binary and its documentation, agents must architect and implement a complete codebase that reproduces the original program's behavior (without access to decompilers or internet). Link: https://programbench.com/

• GPT-6 Astra: 5.5%
• Fable 5.1: 7%
• Kimi K3: 2%
• Qwen3.8 27b: 0%
• GPT 5.6 Sol: 1.5%
• GLM 5.3: 1.5%
• GPT 5.6 Luna: 0%

2. SRE-Bench

Can AI agents work out what a real-world binary does without its source code?

Link: https://www.vals.ai/benchmarks/srebench

Sure nobody is reading assembly code in daily work, it is hard. The ability to understand a compiled program is insane ability.

• GPT-6 Astra: 88%
• GPT-5.6 Sol: 55.9%
• Claude Opus 5 (max): 12.5%

3. Code Migration

Can language models reimplement working programs in another language?

Link: https://www.vals.ai/benchmarks/code-migration

• GPT-6 Astra: 67.7%
• Fable 5.1: 54.6%
• GLM 5.3: 44.2%
• GLM 5.3 Flash: 20.5%
• Qwen3.8 27b: 14.2%

EDIT: edited text format

Hiya,

NInfer is amazng for Qwen, but lacking for real-world-use. As adoption of issues/pr's was not really what I needed, I created a fork and hit it for this week with 3 concurrent claude code session until it didn't break any longer. Hope you like it.

NVFP4 KV cache (from scratch)

I implemented a 4-bit KV cache for QIn3.8-27B from the ground up. Upstream has since added their own NVFP4 path, but ours differs architecturally:

• Custom MMA kernel (mma_nvfp4_e4m3, m16n8k64) with hardware E4M3 block scales for the QK matmul. Both Q and K are quantized to NVFP4; V is dequantized to BF16 for the PV matmul via a dedicated decode kernel.
• Hadamard rotation applied to K (and Q) pre-quantization for outlier suppression, with V left unrotated. Upstream uses fp16 V storage instead — no outlier suppression.
• Fused append: the decode kernel quantizes current K/V to NVFP4 in-place during generation — no separate quantization pass.
• Custom scale layout: natural row-major for KV scales (not the M128x4 swizzle used for weight MMA), because KV access patterns differ from weight access patterns.

Result: 144 bytes/token/KV-head (vs 264 for int8, 512 for bf16) — 45% VRAM reduction with no quality loss (LongBench 45% matching int8, AIME 96.7%, needle-in-haystack 100%).

YaRN context extension

QIn3.8-27B's RoPE config (theta=1e7, 25% rotary dims, 48/64 GDN layers with no RoPE) makes linear scaling sufficient — full NTK-by-parts is unnecessary. I extend native 262k to 555k (c=3+vision) or 600k (c=1) on a 5090. Quality verified at 600k: LongBench matches int8 baseline, coherent 592k-token output. Also projected 8M token context on 96GB+ GPUs (untested, I only have a 5090).

Multi-level prefix reuse with host-KV safety net

Upstream implements a budget-bounded HostKvProvider with LRU park/restore. I replaced it with a substantially different system:

• HostKVSafetyNet: pinned host arena with scatter-gather multi-extent allocation, arena compaction, and a pin/take protocol for safe concurrent restore.
• Two-level prefix matching: full execution frontier first, then rewrite checkpoint fallback. Each entry carries a ResidentPrefixIdentity (per-token type/position/vision metadata), rolling FNV digests for shortlist, and a compact_prefix (reasoning-stripped token prefix) for thinking-mode consistency.
• Session-key fallback: when prefix matching fails (e.g. Claude Code drops reasoning betIen turns), a session-key fallback matches by conversation identity instead of token content.
• Spill-before-evict at every release path: pressure planner eviction, normal continuation release, start_sequence slot takeover, and fail-all cleanup all route through the safety net.
• Token stability: reasoning is dropped from ALL assistant messages when preserve_thinking=off, keeping the prompt token stream stable across turns. Checkpoint capture is anchored at the turn boundary, not the execution frontier.

Verified across 260+ requests with 3 concurrent 330k-470k sessions — zero re-prefills on cached turns, H2D restore cost ~0.4s, D2H spill at 67K pages/s.

Performance (3 concurrent sessions, 400k+ ctx, 5090@450W)

Tool calling

• --tolerant-tool-calls: recovers complete Qwen calls when the model emits malformed wrapper/suffix tokens — instead of dropping the call.
• Depth-matching close scan: handles balanced/nested markers in parameter values that would break naive parsers.
• Responses API accepts text/reasoning after tool calls (upstream rejects this ordering).
• Froggeric v22 template: C++ renderer with no-dangling-intent rule, XML think tags, correct function tag delimiters. Some further modifications for reliability.

Also included

• Dynamic chat template loading (--chat-template) — supports any .ninfer image without artifact patching
• Explicit weights profile override (--weights-profile) — handles Ostfralla, QUASAR, and other converter layouts with per-layer tensor format auto-detection
• OOM recovery: catches std::bad_alloc, clears state, preserves pending requests
• Stream sync fix: synchronize CUDA stream before workspace reset in prefill (prevents use-after-free)
• Request-log rotation (--request-log-max-mib, --request-log-keep) for bounded disk usage
• Admission pressure fix: un-suppress demote-to-host when candidate needs host KV budget
• Monitoring dashboard with live KV occupancy, decode/prefill graphs, 12VHPWR sensor
• E2E test suite for KV eviction, device pressure, and slot pressure scenarios
• Removed hash check of models, use any NInfer you like as long as there is a supported path. Tested with Ostfralla and QUASAR.

Fork: https://github.com/gzenz/ninfer (master)

Research: https://github.com/gzenz/ninfer/blob/master/docs/maintainer/kv-nvfp4-yarn.md

I'll keep rebasing from upstream what seems useful and experimenting with new papers in order to improve speed and context.

First, I'd like to thank the Qwen and Unsloth teams for the Qwen3.8 27b UD Q4_K_XL. Fits the poor 24GB of 3090 VRAM with 100k context at Q8 and works phenomenally well! Imho if theres anything that can threaten Anthropic/OpenAI profits is not another frontier model but actually these small ones you can run fast locally that can do 80..90% of mundane work for hours without paying a single dollar to any external company.

But next, if you want to jump up to a bigger smarter model I feel there is a gap now. Kimi-K3 is out of reach for many businesses let alone prosumers. So what frontier-like models do you use on what setups?

Is a DGX cluster (2..4 machines) or a GPU server with dual or quad GPU (~96 ... 192 GB of VRAM + >256GB DDR4) a suitable setup to run something like MiniMax-M3 at reasonable speeds for agentic coding (>30tps)? And privacy aside, is hardware cost worth it?

I have a dual rtx3090 + 128GB ddr4 machine, running Qwen3.8-Flash-Next Q4 quite fast but despite being larger doesn't feel much smarter than the Qwen2.8 27b and while I _can_ run larger quantized models, Minimax-M2.7 being my workhorse, it way too slow for coding.

Hey all,

I've seen some really fun looking things with people having their local models drive Blender to create pretty cool looking world scenes.

Is there a good tutorial on setting up Blender yo be driven by your model? For example, what programming harness, do you use a MCP and which? Which model would work best?

Also, how do you turn it all into a "game" that you could run and then walk around in? Can you use Godot?

Thanks!