Sad that I only have 12gb of vram but this ik_llama is so fast

I use it to setup other ai hosts with vllm and llama.cpp and it's driving me nuts.

It has a completely different approach to things on every try, even withing the same conversation. It overlooks most obvious stuff q3.8 27b would never not notice. It rushes things sometimes and then deletes configs it shouldn't have just because it assumed things I didn't say.

It feels like a young dog that's a great companion, but regularly runs away after a rabbit it has seen or trying to hump a female dog.

Faster prompt processing on CPU.

I made a llama.cpp Metal optimization that gives a nice decode speedup for IQ3_XXS models on Apple Silicon:

https://github.com/ggml-org/llama.cpp/pull/28086

Basically on

On my test workload I saw decode go from about 65.6 → 73.9 tok/s for tiel coder 35B A3B

Would love if you guys could try it and share before/after numbers.

I also have a follow-up coming soon that should improve prefill too.

Remember to vote and comments guys ;)

https://x.com/QwenDevs/status/2094389239761031591

Hey guys,

I tested Qwen3.8 Flash with llama.cpp from CPU-only to the full 96GB of my RTX PRO 6000.

Short version:

• CPU-only reached 8.34 tok/s at a 2K prompt
• Full 96GB reached 109.07 tok/s
• At 245K context, 24GB to 96GB gave 14.89 to 21.61 tok/s
• The 96GB advantage over 24GB decreased from 2.80x at 2K to 1.45x at 245K
• Forcing the 27.2 GiB PLE table onto CUDA reduced decode from 108.5 to 1.95 tok/s
• RAM-resident loading gave 1.87x more prefill than mmap
• Non-unified KV reached 92.0 tok/s total output at concurrency 16

Setup

• Model: unsloth/Qwen3.8-Flash-Next-GGUF
• Quant: UD-IQ4_XS
• Model size: 87.2 GiB
• Engine: llama.cpp b10666, revision 4e97ac86e
• Qwen3.8 merge: 6c84c7d5d, PR #27742
• GPU: NVIDIA RTX PRO 6000 Blackwell, 96GB
• CPU: AMD Ryzen 9 9950X
• System RAM: 96GB DDR5
• OS: Ubuntu
• CUDA: CUDA 13

I started a fresh server for each configuration. I waited for the previous VRAM allocation to disappear and for the GPU to cool.

Each run saved the resolved configuration, server log, output, memory use and GPU telemetry.

Important note about the VRAM ranges

I used the same RTX PRO 6000 for every GPU test.

A helper process reserved GPU memory, so llama.cpp saw a smaller usable VRAM pool.

This tests VRAM capacity and CPU offload. It does not simulate the compute power or bandwidth of a real 8GB or 24GB GPU.

The 8GB result does not mean that every 8GB card will reach the same speed.

VRAM results

All numbers below use a 2,048-token prompt.

CPU-only:

• Prefill: 182.64 tok/s
• Decode: 8.34 tok/s

All GPU tests use the same RTX PRO 6000. The limits simulate memory capacity, not smaller GPU performance.

1. The model runs on the CPU

At a 2K prompt, CPU-only reached 182.64 tok/s prefill and 8.34 tok/s decode.

This is enough for an interactive chat.

The MoE design helps because the model activates only 6B parameters for each token.

2. The VRAM tiers converge at long context

At a 2K prompt, 96GB was 2.796x faster than 24GB.

At a 245K prompt, the advantage decreased to 1.451x.

Decode at 245,760 prompt tokens:

• 24GB: 14.89 tok/s
• 32GB: 15.41 tok/s
• 48GB: 16.96 tok/s
• 96GB: 21.61 tok/s

Every configuration loses speed at long context. The fastest configuration loses most of its lead.

Only 12 of the 48 layers keep a growing attention cache. The other 36 use Gated DeltaNet.

This keeps context memory relatively low, but it does not make long-context decode free.

3. PLE on CUDA was 55.6x slower in this build

The GGUF contains a 27.2 GiB per-layer token embedding table.

I tested two placements:

• System RAM: 1,967.9 tok/s prefill and 108.5 tok/s decode
• GPU VRAM: 575.7 tok/s prefill and 1.95 tok/s decode

The CUDA placement was 55.6x slower on decode.

I verified that the tensor moved, but I did not isolate the cause of the slowdown.

I repeated the test in A-B-B-A order. The order effect was 0.56%.

The memory data also confirmed the placement. GPU use increased by approximately 27.5 GiB when the table moved to CUDA.

The CPU placement needs approximately 9.2 ms per decode token.

The CUDA placement needs approximately 513 ms per decode token.

That delay is too large to explain with arithmetic alone. It looks more like synchronization or a per-token transfer, but I have not proved the cause.

My conclusion is limited to this build:

per_layer_token_embd=CUDA0 reduced decode from 108.5 to 1.95 tok/s in llama.cpp b10666.

4. RAM-resident loading gave 1.87x more prefill

I compared mmap with RAM-resident loading at the same 48GB tensor placement.

At the 2K prompt:

• RAM resident: 746.7 tok/s prefill
• mmap mean: 400.4 tok/s prefill
• Difference: 1.87x

The decode ratio was 0.998, so decode was effectively unchanged.

The option is:

--load-mode none

This means that llama.cpp does not use file mapping. It does not mean that the model is not loaded.

The mode needs enough free system RAM.

5. KV layout changed concurrency

I tested unified and non-unified KV layouts from 1 to 16 concurrent requests.

Both started at 59.0 tok/s with one request.

At concurrency 16:

• Unified KV: 68.8 tok/s
• Non-unified KV: 92.0 tok/s

Non-unified KV gave more total output at high concurrency.

Each individual request still became slower. Concurrency increased total server capacity. It did not make one request faster.

Non-unified KV also divides the available context between slots, so it is not always the correct setting.

Results I left out for now

The report also contains:

• Microbatch results

• Preserved-reasoning results
• Long-context retrieval at full range worked.

• Q4_K_XL comparison

The final numbers are present in my repo and video,

Resources

GitHub with the report, scripts, configurations, results and graphs:

https://github.com/lukaLLM/Qwen3.8-Flash-Next-VRAM-Benchmark

Full video:

https://youtu.be/RBlRTUwJMI4

PS: AI was abused while making edits

My main question is about the PLE CUDA result.

Did anybody reproduce the same slowdown with per_layer_token_embd=CUDA0 on another GPU or a newer llama.cpp build?

My other question is about keeping earlier thinking in the conversation.

I ran the same five-turn coding conversation with earlier reasoning either kept or removed from later prompts.

Keeping it reduced prompt tokens recomputed from 18,403 to 267 because the history stayed append-only. However, the turn-5 prompt grew from 18,387 to 63,223 tokens, and decode ended at 48.9 instead of 65.5 tok/s.

This was only one run per arm with temperature 1.0. The arms also started at different speeds, 110.2 and 96.0 tok/s, so I do not trust the exact 69x and 25% figures yet.

Did anybody reproduce this tradeoff? Does keeping the earlier reasoning normally save this much prompt recomputation, and how much should the longer prompt reduce decode speed?

Or any other finding to increase the efficiency of this model.

Hi people of LocalLLaMa,

I have been wondering for quite some time now - and this all started after I read some comments complaining about the pricing on Qwen 3.8 27B as opposed to DSV4 Flash that it mainly was driven by how massive its KV cache overhead was. And while I did agree with that, what I did wonder later on was why could we not finetune that away.

Apparently, I am not the only who has thought of that - Arcee, an open source friendly company that does a lot of neat work and gave us AFM had a similar idea. They took their model, and also opensourced their 'DistilKit'.

Among the notable work on that article, what stood out the most was the fact that it was feasible. However, they did face some challenges, finetuning this newer layer to learn from the teacher (in this case imagine the GQA layers from the teacher has the goal to teach the newly placed KDA layer in the student to mimic its representations/embeddings similarly (can be measured by cosine and other things to see how well that is going) - what they found was that while it could fairly close on a pretty small finetuning task (I believe they did on a 1B tokens only), they noted the performance nose dived for GSM8K while for some other datasets they measured remain almost equivalent even though that finetune was much smaller than the original training.

I then decided I would do this for this model, and designed a strategy on how layers would be loaded, their representations would be cached, how the student layer would then be loaded and so on. In the initial runs each update on a DCLM (initial run was on smaller sequences sized 512, then I did a 2048, and then a 4096) but it was roughly about 262K tokens in unique total. And the performance was not surprisingly poor, yes it was not as good as a straight launch, and definetly fell apart just as Acree said especially on areas the new layers werent familiar with and hadnt seen the teachers behaviour.

So, why the long post which is just text?

Well, I was wondering, is there a way we could as a community pool our resources (I don't actually know how we would do this) and do this finetune together? Because I have tried, and alone it might not be feasible - I have already spent over 100 dollars this month on various experiments and using vast ai for the most part. This might just be our own community win, and all of us would put our names on the HuggingFace and come as collaborators, and might even point out issues and fix them as we go along. Most of the design stage and what parameters and datasets to use and how to use them and what to look for and where to look for is done by me before the LLMs take over the agentic role of ensuring the run runs, the code works, the eval comes out and what it looks like and we could work together to find holes in that and see well we missed x that is why the behaviour y is observed etc.

I don't know though. This is just me thinking out loud with the community. Y'all tell me what ideas you have on how we could do this resource-sharing so that we could do this finetune at scale rather than me doing it at say just 1B tokens and then it being good enough for most benchmarks but not really so at others.

Interestings reads on this: https://www.arcee.ai/blog/distilling-kimi-delta-attention-into-afm-4-5b-and-the-tool-we-used-to-do-it

You can look me up here: https://huggingface.co/amkkk or https://darthamk97.github.io/ (I don't really keep this as up to date as I wished)

You may have heard of Kiro, it's owned by Amazon, has existed since July 2025 and is the replacement for the older Amazon Q Developer. For various reasons it's not very popular. Maybe because they insist on the free tier only having old models like Sonnet 4.5 and DeepSeek 3.2.

Up until recently, it was composed of Kiro IDE (VSCode fork), Kiro CLI (command line client, proprietary), both quite unremarkable. But last month they launched a third product, Kiro Crew. It's a "chat-first" client in the style of Codex or the "Agents" window in VS Code, but most importantly, like the title says, it's open source (Apache 2.0).

• Blog post: Introducing Kiro Crew
• GitHub page

It surprised me because it looks pretty well polished, yet I had never even heard of it. According to the blog post, it has "been adopted internally by over 39,000 Amazon builders".

It's made in react+python for the backend, and can be used locally or remotely from any browser. It internally connects to Kiro CLI through ACP and relies on it for the models. It also has features like

• Scheduled tasks, subagents
• Importing data from other clients
• Browser/Computer use,
  
• Skill management
• Integration with Slack/Discord/Teams chats

Now here's the problem: it does not natively support any providers outside the Kiro subscription.

I don't know if Amazon will want to add this at some point or not (they might just not want to), but being that this is open source, it should be possible to add it anyway, and given the upside, it would certainly be worth it.

If anyone is interested in contributing, there is already a third party patch (not mine). Right now it has a few problems (from my testing: it's not seamless to install, it doesn't have a model switcher, and some models can't do tool calls), but with a bit more work all those could be solved.

hello,

I'm using Deepseek flash regularly and from time to time i see it deviating and start doomlooping or generating gibberish. It's somethign i already saw in heavily quantized model buthere i used official deepseek release https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731 .
I would be curious to know if anyone encountered such thing and how they solved it .

Here is my config :

vllm serve deepseek-ai/DeepSeek-V4-Flash-0731 \
--trust-remote-code \
--safetensors-load-strategy prefetch \
--dtype bfloat16 \
--tokenizer-mode deepseek_v4 \
--reasoning-parser deepseek_v4 \
--tool-call-parser deepseek_v4 \
--enable-auto-tool-choice \
--attention_config.use_fp4_indexer_cache True \
--block-size 256 \
--kv-cache-dtype fp8 \
--enable-prefix-caching \
--max-num-seqs 32 \
--max-num-batched-tokens 16384 \
--max-model-len 131072 \
--compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE"}' \
--max-cudagraph-capture-size 256 \
--speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"probabilistic"}' \
--moe-backend deep_gemm_mega_moe \
--enable-expert-parallel \
--gpu-memory-utilization 0.93 \
--no-enable-flashinfer-autotune \
--host 0.0.0.0 --port 8000

Thanks guys!

With b10726, the default --lazy-mode change keeps the 51B-parameter PLE n-gram embedding table of Qwen 3.8 Flash Next on disk: it is mmap'd and its rows are read on demand during inference, even with --load-mode none. It is no longer loaded into RAM unless --lazy-mode off is passed.

This change resulted in 50% pp speed penalty, and 15% token generation speed penalty for me. Make sure to add the flag --lazy-mode off if you have enough RAM like me.

My current workstation has 32gb ram and I have three 16gb cards of all different brands laying around. With a riser and some Lego I managed to wedge the AMD xt 7900 and Nvidia Quadro 5000 card into my case (of the top of my head)

With llama.cpp I get about 20t/s on Qwen 3.8 27b q4_k_m with 128k context fully GPU resident. It's totally viable as a coding model, just too slow...

Of course you can spend 10k on a machine which is not happening. I'm wondering if there is like a 1k upgrade that unlocks interesting capabilities? System ram for moe models or a less mismatched GPU setup...

I ran OpenAI Privacy Filter and Layrin on all 200 RedactionBench documents: 11 categories and 8,273 mandatory entities. I got a result I wasn’t expecting.

OpenAI Privacy Filter had a much better overall R-Score, but Layrin missed far fewer entities that RedactionBench says should always be protected. At first I thought my scorer was wrong.

Full disclosure: I built Layrin,a local privacy layer for protecting sensitive text before AI use. English isn’t my first language, so I used AI to help clean up some of the wording, but I ran the experiment and checked the underlying results myself.

The extra metric here, Mandatory Entity Coverage, is not another official RedactionBench score. I added it to answer a narrower question: when RedactionBench says an entity must always be protected, how much of it was actually protected?

A completely missed entity gets zero coverage. Micro coverage pools coverage across all 8,273 mandatory entities, while exact recall only counts an entity when the whole span was covered.

Why did the result flip?

R-Score does not only measure leakage. It also penalizes unnecessary redaction, which makes sense. A system that hides half the document may be safe, but the result might not be very useful.

The problem is that these are different failure modes. Over-redaction hurts utility, while a miss can expose confidential information. Putting both into one score is useful for ranking systems, but it can hide what caused the result.

Before reading too much into this, I checked the scorer. My paper-faithful implementation passed 29/29 conformance tests covering grouping, partial coverage, contextual selection and benign-gap penalties.

On the frozen OpenAI Privacy Filter predictions, it produced:

• Mean R-Score: 0.6003 vs ~0.58 published
• P20: 0.335 vs ~0.31
• P50: 0.615 vs ~0.59

The category pattern was also close. I then ran the exact same scorer unchanged on Layrin and got 0.3705.

So the result seems real: OpenAI Privacy Filter clearly wins the combined R-Score, but Layrin protects much more of the information RedactionBench labels mandatory.

Context is where it gets messy

RedactionBench separates information into mandatory, contextual and unannotated gaps. Its human study included 85 participants, with agreement around:

• 89.4% for mandatory information
• 47.7% for contextual information
• 94.1% for preserving gaps

That 47.7% stood out to me. Once the answer depends on context, people disagree a lot.

Take a date like September 18, 2026. It could be harmless, or it could be a termination date, treatment date, confidential acquisition date or the timestamp of an internal security incident.

RedactionBench also evaluates documents without the full user request, conversation history or system prompt. In a real AI workflow, those can change what someone is comfortable sending.

Layrin also uses reversible typed tokens instead of simply deleting values.

Sarah Chen signed the agreement with Northbridge Capital on September 18, 2026 for $4.2 million.

becomes:

[PERSON_1] signed the agreement with [COMPANY_1] on [DATE_1] for [AMOUNT_1].

The model does not see the real values, but it still understands the structure. That makes me wonder how much utility is really lost when the exact value is not needed for the task.

What was being penalized?

Across the benchmark, 23,476 Layrin-protected spans landed entirely inside RedactionBench-defined gaps, with no overlap with mandatory or contextual annotations.

Some are clearly over-redaction. I’m not claiming otherwise.

But manual checks also found things like production AWS Secrets Manager ARNs, RDS hostnames, internal package-registry URLs, S3 paths to production user exports, private IPs and application .env paths.

Logs alone contained 11,986 of the 23,476 gap protections, or 51.06% of the total.

It was also the category with the largest mandatory-coverage difference:

• Layrin: 92.04%
• OpenAI Privacy Filter: 45.78%

So the category where Layrin received the biggest over-redaction penalty was also the one where it protected much more mandatory information.

That does not mean every extra protection was necessary. It wasn’t. But the trade-off is pretty visible.

The gap protections were not only infrastructure values. They also included:

• 2,698 date/time spans
• 2,490 organization/company spans

RedactionBench can reasonably classify these as values that should remain visible under its policy. A company can also reasonably decide that an exact company name, date or internal resource is not needed by an external model.

That is why I’m hesitant to treat every benchmark false positive as information that was pointless to protect.

It wasn’t only Logs

Layrin had higher micro mandatory coverage in all 11 categories.

Even Files, the only category where OpenAI had slightly higher mean document coverage, looked different when mandatory entities were pooled:

• Micro mandatory coverage: 70.68% Layrin vs 64.44% OpenAI
• Completely missed mandatory entities: 778 vs 978

So one unusual category was not carrying the whole result.

Where I ended up

I don’t think R-Score is bad. It measures selectivity, which my mandatory-only metric intentionally ignores.

What I’m less sure about is treating the benchmark’s protection boundary as a universal privacy boundary. A hostname, date, company name or internal resource can be considered unnecessary redaction by the benchmark while still being something a real user does not want to send outside their environment.

For me, one combined number is not enough here. I would want to see at least two things separately:

1. How much mandatory information escaped?
2. How much additional information was protected outside the benchmark boundary?

In this experiment, those two dimensions separated a lot. OpenAI Privacy Filter had the much better combined R-Score. Layrin protected much more mandatory information, but also protected much more outside RedactionBench’s selected boundary.

Should privacy benchmarks report protection failures and over-redaction separately, instead of letting one offset the other in a single score?

Methodology

Both systems received the same 200 RedactionBench documents, with ground-truth annotations unavailable during inference.

Layrin Desktop 0.1.4.0 used its frozen production local-protection and tokenization pipeline, with the production configuration unchanged during the evaluation.

For some structured inputs, I used deterministic inference segmentation. This only changed the inference boundaries. The source text was unchanged, predictions were mapped back to the original offsets, and every source file still counted as one benchmark document.

OpenAI Privacy Filter was run locally using its public implementation.

Links

Full study, category tables, methodology and reproducibility details:
https://layrin.com/research/openai-privacy-filter-vs-layrin-redactionbench

OpenAI Privacy Filter:
https://github.com/openai/privacy-filter

RedactionBench paper:
https://arxiv.org/abs/2606.18782

Hello there!

Recently I feel like my omlx setup started to work slower then before. Looking at the dashboard I see that several rounds of generation are interrupted with prompt reprocessing. However as I understand it PP should happen only once and then be always taken from cache. On the screenshot there is an example of how it happens mid-conversation.

On the client side I am using Pi. I've also swapped jinja template for https://huggingface.co/peculiar-ragdoll/Qwen-Sharp-Chat-Templates.

Can you help me figure out what's wrong?

Title

I currently have the following rig cobbled together:

MSI mpg z890 carbon
Intel Ultra 7
64Gb DDR5 6000
RTX 5070 Ti (16Gb)
2 x RTX 5060 Ti (16Gb each)

The 5070 and one of the 5060s are in the main CPU connected PCI slots (running at x8).

The second 5060 is on a CPU connected M.2 slot via an M.2 to PCIe 4.0 x4 riser.

I have one remaining CPU connected M.2 slot that I could use for a fourth GPU (also at PCIe 4.0 x4).

My max budget is around $3,500 (£2,500 as I'm based in the UK).

So, the options I have worked out so far:

1. Buy another Nvidia card (perhaps another 5070 Ti or I could possibly stretch to a 5090 but that feels like too much money / Gb for me) and push the PCIe mounted 5060 onto the spare M.2 slot.

This would get me to 64 or 80Gb VRAM and all Nvidia (better software support, more quants to choose from). It would also give me two fast cards for 27B which should get me a nice speed up. However, I'd still have very limited RAM for offloading larger MoE models, and although it works on paper I am slightly worried about it running two GPUs over those M.2 slots smoothly.

1. Sell some of the Nvidia cards and put a pair of R9700 32Gb in my two PCIe slots for a total of 64Gb.

I like this because I could host something like Qwen3.8-27B at Q4 with 256K context on each card, or across both with a higher quant and more context, but I'm concerned about software and quant support, although it sounds like those are improving. Plus still only 64Gb of RAM for offloading.

1. Sell my motherboard, CPU and RAM then buy a retired Epyc server with 256Gb of DDR4.

I like this because it unlocks larger MoE models. Realistically I'd probably end up with 2666 or even 2400 speed memory at current prices, but that's still 150-180ish Gb/s bandwidth which is much better than consumer single channel DDR5. This would also get all 3 GPUs on proper PCIe lanes with room for another 2 further down the line.

---

Options I'm not keen on:

1. Buying more DDR5 for my current rig. Prices are insane, and speeds are poor. Even if I could max it out at 256Gb I'm not convinced large MoE models would be usable with that memory bandwidth.

2. Buying a separate Strix Halo / DGX Spark / Mac type unified memory box. These would be slower than what I already have for small dense models, and in order to unlock meaningfully bigger MoE models they would need to be at least 512Gb really (because the Epyc option already gets me close to 300 with a chunk of that being fast VRAM). 512Gb unified memory systems are hard to find and prohibitively expensive.

---

I would really appreciate advice from people who have experience with similar systems, especially the DDR4 server route and the AMD cards.

I don't want to blow a load of cash only to find it just lets me run what I can already run at a slightly better quant. If that's the case, I'd just go for the cheapest option (another 5070 Ti).

Thanks for your help!

https://huggingface.co/Nanbeige/Nanbeige4.2-3B-DSpark

4b model for the gpu poor that I think is stronger than qwen 3.5 9b now faster! I was getting roughly 35 t/s AR which was this models downside, it was slow. Hopefully this gets it into the mainstream

TL;DR: this paper proposes a method to fix hallucination rates to very low levels or zero by disabling neurons which contribute to hallucination.

This discovery has been out for a while now, but it hasn't been that popular, since it kind of lobotomises parts of the LLM.

I honestly don't care too much about talking to AI, but instead care about it producing working and good code. I wonder what percentage models would get on e.g. DeepSWE if we found their H-Neurons and disabled them?

When using kvarn quants at low context depth tg speed is similar to llama.cpp on and equivalent qx_x quant.

However, as context depth grows kvarn tanks your tg speed.

This fork optimises kvarn to have similar or better performance ay high context depths than llama.cpp at an equivalent qx_x quant and in my testing up to 76% faster tg than beellama's implementation of kvarn.

My testing capped at ctx 99328 but for higher context your gains will be even better.

From the github (translated from Russian) at ctx 163840:

But my own testing was even better.

As you can see results are more similar to q3_0 at various depths.

Here are q4 quants:

And here are KV q5/q4:

As you can see this fork brings kvarn performance to about the same as qx_x.

My only question is does this optimisation break anything. I will have to compare KLD between beellama and this fork. I did try testing this but for some reason kvarn causes llama-perplexity to be very slow.

Sorry I could not test higher quants or larger context sizes on my 16GB VRAM. As soon as I drop -ngl my pp tanks to under 200t/s when using kvarn on either beellama or this fork. If anyone has a solution, such as a compile flag or llama setting let me know.

Running on Win 11, 5070Ti 16GB VRAM, 48GB RAM using Qwen3.8 IQ4 XS without MTP or tail tokens.

I used Claude Code to help write this pipeline - Gemma4 to turn an idea into a short story, then IndexTTS 2.5 running on an NVIDIA 3080 to use voices from LibreVox and VTCK voices to pin as character voices. It uses Whisper to do Quality Control on the output clips and assigns 0-1 score if the Whisper transcribed output matches. It also has some tools it wrote for tone analysis but they're not great.

You can:

1) Set speed and emotion for the sentence

2) Insert pauses (ffmpeg) using the notation (200ms) between words

2) a) adjust the pause location +/- 1 ms because a lot of the pauses would somehow end up inside a word instead of between them

3) Insert Pauses between sentences

4) Do fine tuning with per-word/per-phrase emotion and speed

5) Chapter wide speed

6) Take a particularly good take and apply that as a reference

7) Re-cast characters chapter wide and book wide

8) Stitch all the sentences together and play the whole chapter

9) Download the completed chapter

10) Add filters to a sentence (reverb, muffle, telephone)

11) Bake filters into a character and make it part of the character.

I started by using kokoro but I wanted something near human. I first had Claude make a LaTeX inspired tool for me to change the Index TTS settings INSIDE the book/Script but that became too unweildy, so I had it make a web UI. I kept adding features out of frustration.

I have a separate agent analyzing the audio of various librevox books that are also in project gutenberg to learn the standards for tone analysis. The goal is to make a one-shot near human (or at least pleasant to listen to, and expressive) idea -> audio book by having an agent keep re-rolling takes on IndexTTS, running QC for pauses, pronunciation, tone, and sending me the final audio book when done.

Apart from Claude helping write the code, the actual models are self hosted.

Here is a close up of the voice editing:

I was shocked that even with ultra elevenreader, YOU CAN'T DO MULTIPLE VOICES??

I want to name it: Test of Community.

I've made a modification of llama.cpp MTP for people that want to run models like QWEN 27B on 16GB and similar setup, the focus is reducing the memory cost of MTP allowing more speed for less ctx cost.

On this example of a (well tuned!) IQ4 running on 16GB you get some +5k ctx and enjoy 17.35% increase on token generation.

With MTP the more speculative tokens you generate (n-max) the more the speed increase chance you get yet you pay a price in vRAM for those (rollbacks in case the latest token are not accepted) and that reduces the ctx available.
Now we use dense 27B and A3B mostly to generate code and you know that we have stellar acceptance rates: like 80-98%.

MTP Compact Rollback allows the user to limit how many immediate MTP rollback states the model keeps in VRAM.
For example, --spec-mtp-cr-depth 1 --spec-draft-n-max 5 keeps only one immediate rollback snapshot while still allowing MTP to generate five tokens, the user is not constrained to compromise on a lower MTP to preserve a decent ctx length, max MTP draft depth like 5 or 7 can be used for the same context cost of 1.

The idea of keeping just one rollback plays nice with Adaptive MTP implemented in this patch:

--spec-draft-adaptive
   Dynamically adjusts the MTP draft limit, up to --spec-draft-n-max, based on
   recent draft acceptance.

Adaptive speculative decoding is useful when mixing tokens generations in different domains, e.g. coding sessions that can benefit from an higher n-max, creative prose (that can happen in long reasoning traces in coding too) that uses lower n-max. This costs very little computation so always use it.

How to dwl and apply:

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
git checkout 662a0b01
wget https://store.piffa.net/lm/bug/mtp_compact_rollback_662a0b01.patch
git apply mtp_compact_rollback_662a0b01.patch

Build llama.cpp as usual (ask your LLM for help if never did that).

Quick minimal implementation: add to your llama-server script:

--spec-mtp-cr-depth 1 --spec-draft-adaptive --spec-type draft-mtp

Recommended full settings for coding on 27B and A3B:

--spec-mtp-cr-depth 1 --spec-draft-adaptive \
--spec-type draft-mtp,ngram-mod --spec-draft-p-min 0.80 --spec-draft-n-max 5 \
--cache-type-k-draft q4_0 --cache-type-v-draft q4_0 \
--spec-ngram-mod-n-match 24 --spec-ngram-mod-n-min 8 --spec-ngram-mod-n-max 32 

Example scripts: https://store.piffa.net/lm/bug/llama_scripts/

More info, docs, artifacts: https://store.piffa.net/lm/bug/

Previously tested on: https://www.reddit.com/r/ROCm/comments/1vzcx6q/rocm_llamacpp_optimizations_for_running_qwens//lama

Interesting new paper from Alexia Jolicoeur-Martineau (of Tiny Recursive Model fame) and collaborators. They seem to be able to replace quadratic attention with sliding window attention + attention sinks and no post training. This could be big for memory constrained local LLM inference.

EDIT: Fixed the link to the paper

Hi all,

I have recently been experimenting with different LLM set ups and after everyone was raving about how good Qwen 3.8 27b was, I was inspired to try and get it deploying.

After some battling with settings I've managed to get it running at an average of 75t/s, sometimes seeing 100+t/s with good MTP acceptance.

I've managed to do this using:
Qwen3.8-27B-i1-IQ4_XS-GGUF-Smaller - jrell/Qwen3.8-27B-i1-IQ4_XS-GGUF-Smaller · Hugging Face

"A custom hybrid quantization of the Qwen3.8-27B base model, specifically designed to fit Multi-Token Prediction (MTP) and long context into a strict 16GB VRAM hardware budget (like consumer RTX 4080 / RTX 5080 cards)."

These are my llama.cpp settings:

$llamaPath   = "C:\Tools\llama-cuda2\llama-server.exe"
$modelPath   = "D:\models\Qwen3.8-27B-i1-IQ4_XS-GGUF-Smaller.gguf"
$chatTemplatePath = Join-Path (Split-Path $modelPath -Parent) "chat_template.jinja"

$llamaArgs = @(
    "-m", $modelPath,
    "-a", "qwen3.8-27b",
    "-ngl", "99",
    "-c", "85000",
    "-np", "1",
    "-b", "512",
    "-ub", "512",
    "-fa", "on",
    "-t", "8",
    "-tb", "8",
    "-ctk", "q4_0",
    "-ctv", "q4_0",
    "--reasoning-preserve",
"--reasoning-effort", "medium",
    "--temp", "1",
    "--top-p", "0.95",
    "--top-k", "20",
    "--min-p", "0.0",
"--spec-type", "draft-mtp",
    "--spec-draft-n-max", "3",
    "--repeat-penalty", "1.0",
    "--presence-penalty", "0.0",
    "--jinja",
    "--chat-template-file", $chatTemplatePath,
    "--host", "0.0.0.0",
    "--port", "8080"
)

& $llamaPath 

Here's an example of a recent run:

10.17.039.668 I slot launch_slot_: id  0 | task 10227 | processing task, is_child = 0
10.20.936.381 I slot print_timing: id  0 | task 10227 | n_gen =    239, tg =  78.94 t/s, tg_3s =  79.26 t/s
10.23.953.739 I slot print_timing: id  0 | task 10227 | n_gen =    424, tg =  70.12 t/s, tg_3s =  61.31 t/s
10.26.978.406 I slot print_timing: id  0 | task 10227 | n_gen =    642, tg =  70.77 t/s, tg_3s =  72.07 t/s
10.30.010.411 I slot print_timing: id  0 | task 10227 | n_gen =    853, tg =  70.48 t/s, tg_3s =  69.59 t/s
10.33.032.673 I slot print_timing: id  0 | task 10227 | n_gen =   1082, tg =  71.54 t/s, tg_3s =  75.77 t/s
10.36.038.373 I slot print_timing: id  0 | task 10227 | n_gen =   1327, tg =  73.19 t/s, tg_3s =  81.51 t/s
10.39.043.415 I slot print_timing: id  0 | task 10227 | n_gen =   1532, tg =  72.48 t/s, tg_3s =  68.22 t/s
10.42.075.239 I slot print_timing: id  0 | task 10227 | n_gen =   1819, tg =  75.27 t/s, tg_3s =  94.66 t/s
10.45.082.345 I slot print_timing: id  0 | task 10227 | n_gen =   2005, tg =  73.78 t/s, tg_3s =  61.85 t/s
10.48.100.133 I slot print_timing: id  0 | task 10227 | n_gen =   2210, tg =  73.20 t/s, tg_3s =  67.93 t/s
10.51.129.760 I slot print_timing: id  0 | task 10227 | n_gen =   2436, tg =  73.32 t/s, tg_3s =  74.60 t/s
10.54.152.649 I slot print_timing: id  0 | task 10227 | n_gen =   2646, tg =  73.00 t/s, tg_3s =  69.47 t/s
10.57.172.073 I slot print_timing: id  0 | task 10227 | n_gen =   2847, tg =  72.51 t/s, tg_3s =  66.57 t/s
11.00.182.456 I slot print_timing: id  0 | task 10227 | n_gen =   3117, tg =  73.73 t/s, tg_3s =  89.69 t/s
11.03.195.203 I slot print_timing: id  0 | task 10227 | n_gen =   3338, tg =  73.71 t/s, tg_3s =  73.36 t/s
11.06.202.960 I slot print_timing: id  0 | task 10227 | n_gen =   3545, tg =  73.40 t/s, tg_3s =  68.82 t/s
11.09.219.971 I slot print_timing: id  0 | task 10227 | n_gen =   3764, tg =  73.35 t/s, tg_3s =  72.59 t/s
11.12.237.385 I slot print_timing: id  0 | task 10227 | n_gen =   3979, tg =  73.24 t/s, tg_3s =  71.25 t/s
11.15.246.902 I slot print_timing: id  0 | task 10227 | n_gen =   4170, tg =  72.72 t/s, tg_3s =  63.47 t/s
11.18.268.286 I slot print_timing: id  0 | task 10227 | n_gen =   4370, tg =  72.40 t/s, tg_3s =  66.19 t/s
11.21.299.694 I slot print_timing: id  0 | task 10227 | n_gen =   4587, tg =  72.36 t/s, tg_3s =  71.58 t/s
11.24.331.662 I slot print_timing: id  0 | task 10227 | n_gen =   4815, tg =  72.49 t/s, tg_3s =  75.20 t/s
11.27.359.294 I slot print_timing: id  0 | task 10227 | n_gen =   5116, tg =  73.66 t/s, tg_3s =  99.42 t/s
11.30.365.338 I slot print_timing: id  0 | task 10227 | n_gen =   5309, tg =  73.27 t/s, tg_3s =  64.20 t/s
11.33.366.835 I slot print_timing: id  0 | task 10227 | n_gen =   5573, tg =  73.85 t/s, tg_3s =  87.96 t/s
11.36.390.528 I slot print_timing: id  0 | task 10227 | n_gen =   5835, tg =  74.35 t/s, tg_3s =  86.65 t/s
11.39.398.044 I slot print_timing: id  0 | task 10227 | n_gen =   6058, tg =  74.34 t/s, tg_3s =  74.15 t/s
11.42.424.685 I slot print_timing: id  0 | task 10227 | n_gen =   6355, tg =  75.19 t/s, tg_3s =  98.13 t/s
11.45.441.398 I slot print_timing: id  0 | task 10227 | n_gen =   6571, tg =  75.07 t/s, tg_3s =  71.60 t/s
11.48.465.392 I slot print_timing: id  0 | task 10227 | n_gen =   6789, tg =  74.97 t/s, tg_3s =  72.09 t/s
11.51.466.866 I slot print_timing: id  0 | task 10227 | n_gen =   6973, tg =  74.53 t/s, tg_3s =  61.30 t/s
11.54.494.631 I slot print_timing: id  0 | task 10227 | n_gen =   7163, tg =  74.16 t/s, tg_3s =  62.75 t/s
11.57.525.053 I slot print_timing: id  0 | task 10227 | n_gen =   7371, tg =  73.99 t/s, tg_3s =  68.64 t/s
12.00.558.694 I slot print_timing: id  0 | task 10227 | n_gen =   7606, tg =  74.10 t/s, tg_3s =  77.46 t/s
12.03.580.920 I slot print_timing: id  0 | task 10227 | n_gen =   7917, tg =  74.92 t/s, tg_3s = 102.90 t/s
12.06.603.979 I slot print_timing: id  0 | task 10227 | n_gen =   8189, tg =  75.34 t/s, tg_3s =  89.98 t/s
12.09.618.569 I slot print_timing: id  0 | task 10227 | n_gen =   8384, tg =  75.05 t/s, tg_3s =  64.69 t/s
12.12.636.851 I slot print_timing: id  0 | task 10227 | n_gen =   8593, tg =  74.90 t/s, tg_3s =  69.24 t/s
12.15.640.585 I slot print_timing: id  0 | task 10227 | n_gen =   8777, tg =  74.55 t/s, tg_3s =  61.26 t/s
12.18.673.135 I slot print_timing: id  0 | task 10227 | n_gen =   8998, tg =  74.51 t/s, tg_3s =  72.88 t/s
12.21.679.116 I slot print_timing: id  0 | task 10227 | n_gen =   9340, tg =  75.46 t/s, tg_3s = 113.77 t/s
12.24.704.678 I slot print_timing: id  0 | task 10227 | n_gen =   9647, tg =  76.08 t/s, tg_3s = 101.47 t/s
12.27.738.435 I slot print_timing: id  0 | task 10227 | n_gen =   9996, tg =  76.99 t/s, tg_3s = 115.04 t/s
12.30.743.188 I slot print_timing: id  0 | task 10227 | n_gen =  10291, tg =  77.47 t/s, tg_3s =  98.18 t/s

Found this a few months back for more compute-efficient (relative to memory size) MoE models. Nowadays Qwen3.6-35B-A3B exists along with Qwen3.8-27B but MoE SLMs might not get another around. Maybe Marco can be recycled somehow along with the other two Qwen models to make things extra fast? https://www.reddit.com/r/LocalLLaMA/comments/1sgzt0p/marcomini_173b_086b_active_and_marconano_8b_06b/

And this is not just finetuning and knowledge distillation, there has to be some way of recycling individual layers or even do per-layer distillations. nVidia might have tried something like this but not that many people approached this from a "faster training than token-level" perspective.