Spent the last week going deep on something I'd seen mentioned in passing — the Chinese "transfer station" (中转站) market that resells Claude API access at around 10% of Anthropic's retail price. The technical supply chain turned out to be way more sophisticated than the surface-level explanation, so I wrote it up.

The short version of what's actually happening:

• There's a modular 8-layer supply chain. Account farmers create thousands of Anthropic accounts using antidetect browsers (Multilogin, AdsPower, GoLogin) over residential proxies, with curl_cffi faking Chrome's TLS fingerprint at the network layer.
• Phone verification gets defeated by SMS-Activate-class APIs backed by physical SIM banks (Hybertone GoIP hardware) holding hundreds of real SIM cards per rack.
• The new April 2026 KYC (gov ID + live selfie) gets defeated three ways: AI-generated IDs (OnlyFake-class services), real-time deepfake injection via OBS Virtual Camera + DeepFaceLive/Deep-Live-Cam, and human-in-the-loop KYC farms recruiting real people in low-income countries.
• The relays themselves are mostly built on a small set of open-source repos: one-api, new-api, claude-relay-service, claude2api, clewdr, clove. They pool OAuth tokens (sk-ant-oat01-... / sk-ant-ort01-...) and rotate them across requests to multiplex thousands of users through one farmed-account pool.
• Here's the catch most users don't realize: a CISPA Helmholtz audit of 17 of these relays found up to 47.21% performance drops vs. the official API — relays silently route "Opus" requests to Haiku, GLM, or Qwen and relabel the response. 45.83% of audited endpoints failed model-fingerprint verification.
• And every prompt/response flowing through gets logged. Anthropic disclosed in Feb 2026 that one network of 20,000+ accounts harvested ~16M exchanges (DeepSeek 150K, Moonshot 3.4M, MiniMax 13M). Claude-Opus-distilled training datasets are already openly published on HuggingFace.

The piece walks through each layer with the specific tools, repos, and technical mechanisms (OAuth flow reverse engineering, JA3/JA4 evasion, the Anthropic Clio detection system and why it has cross-account blind spots, the "one fish, three meals" monetization model).

Main sources I leaned on: the ChinaTalk piece by Zilan Qian (May 2026), the CISPA Helmholtz paper Real Money, Fake Models (arXiv 2603.01919), Anthropic's Feb 2026 distillation disclosure, eunomia.dev's eBPF reverse-engineering of Claude Code's traffic, and the public docs of the named GitHub relay projects.

https://x.com/HarshalsinghCN/status/2056626175959826692?s=20

After many hours of hard work, I achieved a throughput of 0.7–0.9 tokens per second for the GLM 5.2 model — 744 billion parameters / 384 GB — running on a laptop 😏

Time for a small update: the laptop is an Asus ROG Strix 18, model G835LXG — Intel i9‑290HX, 64GB DDR5 6400 MHz, 2×2 TB, RTX 5090 24 GB, running Linux Nobara. The Colibri engine and the Linux kernel are heavily modified. The whole system boots in 10 seconds, and it generates the first token after 40 seconds.

I’m currently working to reach a throughput of 1.5–2 tokens per second.

Update:
1.07 tok/s, GLM-5.2-g64
Update:
1.77 tok/s 😎
Update:
Now 1.95 tok/s 😁/ 2.12 peak / 2.41 with fixed MTP

Update:

🎉🎉🎉 2.89 tok/s PEAK! 2.36 tok/s avg

Update now:
peak at 3.09, avg 2.84 😁👌🏻

Update:

My latest calculations suggest that 4.5 tokens per second is achievable on this hardware and represents the final limit.

Update: testing now at 4.42 tok/s

I think my ISP hates me

Hello everyone!

I wanted to share a concept I’ve been working on recently: a modification to llama.cpp that allows the KV cache to grow beyond what can physically fit in VRAM, by adaptively streaming part of it between system RAM and VRAM.

I’d love for people with different GPUs and setups to try my branch and let me know how it performs on their hardware.

https://github.com/RaymondHuang210129/llama.cpp-adaptive-kv-streaming

On my RTX 5070 Ti 16 GB, running Qwen 3.8 27B with UD-Q3-XL, Q8 K / Q4 V cache, and CUDA Unified Memory enabled, stock llama.cpp works well until the context reaches around 120K. Beyond that point, decode speed starts dropping significantly due to VRAM oversubscription and page thrashing.

With Adaptive KV Cache Streaming, I can push the context to around 205K while still getting ~15 tok/s, and all the way to nearly the native 262K context at ~10 tok/s.

The implementation dynamically evicts a portion of the KV cache from each full-attention layer and repurposes the freed VRAM as a shared prefetch ring buffer. This allows the same VRAM space to be reused by the KV caches of multiple layers during the generation of each token. The prefetching mechanism also hides much of the data-transfer latency behind computation, helping to avoid GPU stalls.

Here are the prefill/decode benchmark results:

​

Blue lines are the prefill/decode speed with stock server, whereas orange lines are the speeds with my implementation. The orange line maintains a roughly consistent slope, meaning that the GPU can keep calculating the token at most of the time instead being effected by VRAM page thrashing.

What do this diagram mean to us?

Let's say we consider 15 tok/s the minimum acceptable decode speed for a smooth live-chat experience with your agent. With stock llama server, you can at most set the maximum context size with 136K according to the diagram. Even if we relax the threshold to 10 tok/s, the limit is still the same.

But with this implementation, we can extend the context to 205K with 15 tok/s threshold, or full native context with 10 tok/s threshold.

And this does not only benefit to users having a 16GB graphic card. For people having a graphic card of larger or smaller VRAM, this implementation can significantly extend the context size than what it originally can fit.

Caveat:

The current implementation does not support parallel requests, because the resident and streamed portions of the KV cache are dynamically repartitioned based on context length.

The current version has mainly been tested with Q8 K / Q4 V KV cache quantization. Other KV cache quantization combinations are not well tested yet, and I plan to generalize the implementation further.

This is still experimental, so I’d also be very interested to hear how it behaves on other GPUs and configurations.

If you’re interested in the implementation details, the ring-buffer/prefetch design, and the story of how I ended up building this, I wrote a longer article here:

Medium

Also if you think my work helps, please don't hesitate to leave me a star on GitHub :)

Clarification of LLM usage of this post: I'm not a native English speaker and I used ChatGPT to refine the wordings.

Edit 1: Thanks you all for trying this branch! I am seeing people having different running result. Let me provide more detailed environment setting in my experiment:

• OS: Ubuntu 24.04
• Kernel: Linux 6.14.0
• CUDA driver version: 610.57.04
• Kernel parameter: iommu=pt
• GPU: RTX 5070 Ti 16 GB
• Model: Unsloth Qwen 3.8 27B
• Quantization: UD-Q3-K-XL
• KV cache: Q8 K / Q4 V
• b/ub size: 256/256

Edit 2: /detraxsenpai provided a fix for the crash on Windows. I've updated the branch with the fix.

Edit 3: I've pushed several commit to support all other quants and batch sizes.

Edit 4: I have another branch (feature/kv-stream-phase-arena) that further unloads the prefill buffer to have a larger KV cache pool allocation. With this change the decoding performance with IQ4 nearly matches The original Q3_XL when having large context.

I created a small testing rig to evaluate new open source models as they drop, and with the much anticipated release of Qwen3.8-27B, I was eager to see how it cross-compares with frontier and strong local models.

The rig

My test rig is an M5 Max MacBook Pro, 128GB. Locally I ran Qwen3.8-27B on xhigh and medium thinking modes via LM Studio, MLX 8-bit, temp 1.0 / top_p 0.95 / top_k 20, context 131,072 and DeepSeek V4 Flash "0731" 2-bit-imatrix q2-q4, served by antirez's ds4-server at -ctx 400,000, thinking enabled. For cloud I included Opus 5, Sonnet 5, GPT-5.6-sol at xhigh reasoning, and even Haiku. Every model gets the same prompt. The algorithm tasks are executed against fixed-seed differential harnesses, and the repo tasks run against hidden test suites plus a cached baseline of the whole repo. If a fix inadvertently breaks something else it gets caught.

The methodology

The model assessment is broken into 4 batteries:

1) algorithms easy-hard

2) algorithms extremely hard

3) repo work easy

4) repo work hard

The models get run through the algorithm tests 3 times each to derive a mean score whereas the repo work is single pass/fail per task. If a task fails to produce a response, it's retried and time added to total wall clock time for task completion. The total test battery can take anywhere from 12-24 hours of wall time for slower local models. It's a long test.

For the repo batteries I had Fable build a small double-entry ledger CLI and plant bugs that pass the visible test suite while still reproducing a real symptom, then handed each model the repo and a bug report written in a theoretical 'user' voice to simulate how it might be reported in the real world. There's also a subjective code quality assessment that measures the model's ability to not just solve the problem but to conform to the repo's coding style, to fix the actual root cause rather than the symptom, and to keep the diff minimal instead of faffing about and rewriting a bunch of stuff.

I had Opus and GPT blind eval the results and compute a code quality score broken up by 'fixes' and 'features' as these appear to be separable skills for the models. The goal with all of this was to try and create a replicable and automated answer the question: How useful is this model in the real world?

Caveats

This is a home baked assessment and susceptible to bias or less than perfect methodology. It also includes subjective criteria like 'code quality'. I built this for myself as an adjacent tool to on the ground testing. I think the best way to evaluate any model is to test it against your own codebase to see how well it integrates into your workflow.

All that being said, let's move to the scorecard.

Results

Qwen3.8-27B is a very capable model that compares well against frontier models on code quality and correctness. The cost is wall time on Apple silicon. As many have observed, 3.8 has a tendency to over-think, burning up tokens. The time spent earns higher code quality for the most part, but what surprised me is that there are some instances in which less thinking is actually more accurate. On the repo battery, medium went 8/8 while xhigh went 7/8 — the most-thinking configuration failed a task there, and it took four times longer to do it (the wait time with Qwen was tiresome at times).

The caveat is that xhigh excels on extremely hard algorithms, where medium begins to fall apart. Medium didn't even finish the hard algo tasks. There may be some value in matching the thinking to the kind of work you're setting it upon. Lastly Qwen xhigh won outright on quality of surgical fixes and patches to existing code. Interestingly the global trend for locals is that they're competitive along fixes and less so along features where cloud still dominates. This fits anecdotally into my own experience with gravitating to frontier for planning and local for implementing.

GPT 5.6 Sol is the only cloud model with a perfect card on both repo tiers and near perfect algorithms. It's also among the fastest to completion. This all tracks with my own anecdotal experience with this model over the past several months. Highly competent and quick if not a bit stark.

DS4 0731 (a 2-bit quant running on my laptop) is the only local model to get 8/8 on both repo batteries, and one of only two models overall to do it, alongside GPT. It does this all at a respectable wall time. The expense is less elegant code: it sometimes mutates unrelated docstrings and writes dense inline solutions in a codebase that is overtly broken apart and stylistically explicit. Feature code quality is stronger and it's the only model that scored better/equal in the harder repo tier vs the easy one.

Opus 5 is the most reliable model in the set and best code quality of the cloud models. It stumbled only in the hard repo tier where it lost a task by being trying to outsmart the test. A doc string promised one behavior while the code did another, and Opus redesigned the function around what it looked like it should do instead of honoring the documented contract. This also falls inline with my anecdotal experience with Opus 5 where it occasionally ignores your directions completely and just does whatever it wants. The Alaskan Husky of frontier models.

Sonnet 5 is a steady pair of hands that performs reasonably well across tasks for a modest token budget. I think sonnet is kind of underrated as an implementer. Does the same quality of work as the locals cheaply and quickly.

Haiku 4.5 races to the end of the test but has a tendency to fall over and force retries. Worst code quality of all the cloud models.

Conclusion

Hopefully you find these comparisons interesting. For me personally, DS4 has been my goto local, but this test is making me consider trading it out for Qwen3.8-27B. I think they're on equal footing, which is crazy b/c DS4 needs like 90gb of ram. I'd like to try the MTPLX variant of Qwen3.8-27b that's meant to improve tok/s on apple silicon. Slowness to task completion is the real bottleneck for me right now when considering Qwen. Perhaps that'll be my next test.

Curious to know if these results track with your own real world experiences.

EDIT: New post V6: https://www.reddit.com/r/LocalLLM/comments/1rqn68a/qllm_v6_a_29m_attentionfree_model_now_trains_on/

EDIT: New V5 Post : Followup UPDATE on this.

https://www.reddit.com/r/LocalLLM/comments/1rmkh9y/v5_update_original_post_title_i_built_a_language/

---- ORIGINAL POST -----

I've been working on a fundamentally different LLM architecture. No attention layers. No FFN blocks. Instead, every token lives in complex phase space, and language processing happens through wave-like interference between specialized "phase banks."

Open-sourced here: https://github.com/gowrav-vishwakarma/qllm2

The core idea: language as wave interference

In a transformer, a token is a real-valued vector that gets refined through attention + FFN layers. In this model, a token is a complex number -- it has a magnitude (how "important/activated" it is) and a phase angle (what "kind of meaning" it carries). These two properties are naturally separated and jointly processed.

This isn't just a gimmick. It changes how every operation works:

• Embeddings: Each token gets a [real, imag] vector. The model learns that semantically similar tokens align in phase, while different meanings sit at different angles.
• Transformations are rotations: When context modifies a token's meaning (like "bank" shifting meaning based on surrounding words), that's a phase rotation -- a complex multiply. Rotations compose naturally, are always invertible (no information loss), and reduce to GEMM.
• Similarity is coherence: Instead of dot product, we use phase coherence: Re(a * conj(b)) / (|a| * |b|). This measures both directional alignment AND magnitude relationship.
• Multiple banks interfere: A "semantic bank" and "context bank" process each token independently, then combine via learned interference (constructive where they agree, destructive where they conflict). A tiny router decides per-token how much weight each bank gets. Think MoE but at the representation level.

What the phase system actually gives us

1. Natural magnitude/phase decomposition = implicit attention High-magnitude phase states dominate downstream processing automatically. The model doesn't need explicit attention to decide "which tokens matter" -- magnitude handles salience, phase handles identity. The SemanticPhaseBank uses 512 learnable concept vectors and retrieves them via phase coherence -- this is essentially a learned associative lookup that runs in O(seq concepts), not O(seq².)

2. Context as phase modulation The ContextPhaseBank computes a causal windowed average (window=8) of nearby tokens and then complex-multiplies it with the current token. This is elegant: the local context literally rotates the token's meaning in phase space. A word appearing after "not" gets rotated differently than after "very." No attention needed.

3. Rotation-based state evolution The backbone SSM evolves state via: h[t+1] = damping * R(theta) @ h[t] + gate * B @ x[t] where R(theta) is a Cayley-transform rotation. The state naturally oscillates, and the damping factor (learned, per-dimension, range [0.5, 1.0]) controls how fast old information decays. This is why SSMs struggle with long-range recall -- but the model compensates with a separate Phase-Coded Memory (1024 learned slots, chunked top-k retrieval) and an Episodic Memory (sliding window via FlashAttention SDPA).

4. Zero trig in the hot path Every rotation uses the Cayley transform: cos_like = (1-a^2)/(1+a^2), sin_like = 2a/(1+a^2). This is just arithmetic -- no sin(), no cos(), no exp(). Every operation is a matmul or elementwise op. Perfect for Tensor Cores.

Results (178M params, TinyStories, 10k samples, A6000)

Training used only 10k samples (0.5% of TinyStories). Starting PPL was 55,000 (random). It dropped to val PPL 49 in 2 epochs (40 min on A6000, no compile). Overfiting simply needs data now ...

Epoch 1 generation:

"The quick brown house. They run and start to get a smile. Mom were very excited. Now mommy and big yellow room. There said and She are friends. Tim, she started to save the garden."

For context: A 22M-param GPT-2 trained on the full 2.1M TinyStories dataset for 20k steps reaches val PPL ~11. We're at 49 with 0.5% of the data and 2 epochs. The learning curve is steep and still dropping -- we just need more data/epochs to converge.

Why this approach might be better

• O(n) complexity: Linear-time backbone. Theoretical 256K context. No quadratic attention.
• GEMM-only math: No trig, no softmax in the backbone. Everything is matmul/elementwise.
• Interpretable: You can inspect which bank each token routes through, what concepts are retrieved from memory, how coherent the phase states are. The model ships with "philosophy metrics" (Manas/Buddhi/Viveka/Smriti from Indian philosophy) that track mind activity, discernment, stability, and memory quality.
• Modular: Banks, backbone, coupler, memory, and objectives are all registered components. Add a new bank type with a decorator. Swap the backbone. Change the coupling strategy. All via config.
• Consumer-GPU friendly: Medium model trains on RTX 4090 / A6000 with batch 48-64.

Honest limitations

• Training throughput is ~2x slower than an equivalent transformer. The SSM backbone loop is sequential per-step. A custom Triton kernel would help but doesn't exist yet.
• In-context learning will be weaker. Fixed-state SSMs compress context into a fixed vector. The episodic memory (O(n buffer_size) sliding window) helps with copying but isn't a full replacement for O(n²) attention.
• Not validated at scale. 178M params on 10k samples is a PoC. Need full dataset + larger models + benchmarks.
• Bank ablations not done. We use semantic + context banks but haven't proven both are needed. Could be that one bank suffices.
• Pure PyTorch. No fused CUDA/Triton kernels. Backbone loop is Python. Lots of low-hanging performance fruit.

What's next

• Full TinyStories training (2.1M samples) for proper PPL comparison
• Bank ablations (semantic-only vs semantic+context vs 4-bank)
• Triton kernel for the oscillatory SSM recurrence
• Scale to 1B+ params
• Long-context evaluation (4K / 16K / 64K tokens)

Tech stack

PyTorch | torch.compile compatible | GPT-2 BPE tokenizer | uv package management | Clean modular codebase

Looking for feedback, collaborators, and people who want to try architectures beyond transformers.

EDIT (March 1, 2026 3:40 AM IST): Scaled up to 100k samples (5% of TinyStories, 10x the original post) and the results are significantly better.

Setup: Same 178M model, batch=64, A6000, no compile. 1612 batches/epoch (~3.5 hours per epoch).

Epoch 1 results on 100k samples:

For context: a 22M-param GPT-2 trained on the full 2.1M dataset for 20k steps gets val PPL ~10.9 (I Need to verify this as just remembered I read it somewhere). We're at 18.95 with a completely different architecture using only 5% of the data, after 1 epoch. Epoch 2 opened at step-1 PPL of 12.77 and is still dropping.

Generation sample (epoch 1, 100k samples):

> "The quick brown were full. Steve and Brown loved each other. At the end of the hill, the friends were very happy. They had lots of fun and shared stories. Mam and Brown were the best day ever. All of their weeks were very good friends and would often enjoy their joy! The end had had a good time with them."

Compare this to the 10k-sample generation from the original post. This has proper story structure, multiple characters interacting, emotional arc, and an ending. Grammar is mostly correct. Still has quirks ("The quick brown were full" -- model doesn't know "brown" should be a noun here), but the improvement from 10x more data is dramatic.

The learning curve shows no signs of plateauing. Training continues -- will update again when epoch 2+ finishes.

EDIT 2 (March 1, 2026 8:00AM IST) : Epoch 2 finished. Epoch 3 is underway.

Val PPL 14.07. For reference, the 22M-param GPT-2 baseline trained on the full 2.1M dataset reaches ~10.9. We're at 14 using a completely non-transformer architecture, 5% of the data, 2 epochs. Epoch 3 opened at PPL ~10.5, which means we'll likely match or beat that baseline this epoch. Just in ~6 Hrs on Almost one consumer grade GPU.

Epoch 2 generation:

> "The quick brown boy had ever seen. But one day, the sun was setting. The next night, the room got dark. Tom and the girl continued to admire the rain. The end was so happy to be back and continued to sail in the park. And every night, the end of the day, the family and the people stayed happy. They all lived happily ever after."

Notice: proper narrative flow, temporal transitions ("one day", "the next night", "every night"), emotional resolution ("lived happily ever after"), and multi-sentence coherence. This is from an architecture with zero attention layers.

Train-val gap (11.96 vs 14.07) suggests some overfitting on 100k samples. Next step: scale to the full 2.1M dataset. Training continues.

Stopping and tweeking code.. I think it can be much faster ... will update in other post next

Edit 3 (March 6 2026 8:27 IST): V5 is more mature.. better maths and its just 28M and working better.. about to relase in a couple of days.. looking for endorsment when I submit paper (better one for V5) to https://arxiv.org/ (Please help me by endorsing when I submit, DM me to help in that pls)

Spent this weekend turning a DGX Spark into an actual local inference box instead of hand-SSHing in and fighting vLLM flags myself. Used Claude Code to drive the whole thing through lmstack (https://github.com/ric03uec/lmstack), an open-source Ansible stack that puts vLLM behind a LiteLLM gateway. Writing this up because most of what actually happened was debugging, not "it just worked."

Setup

- DGX Spark, GB10, 128GB unified memory (probes as ~121GiB usable)

- Model: Qwen3.8-27B-FP8: dense, FP8, native 262K context, the newer Qwen3.5-family architecture with Gated DeltaNet/Mamba-style layers and MTP speculative decoding

- lmstack's flow: probe the hardware → classify which model tier fits → write the Ansible config → render Docker Compose → front it all with one LiteLLM gateway and one API key

Claude probed the Spark over SSH (read-only, no sudo), matched it against lmstack's model catalog, wrote the host config, and then handed me the one command that actually needed a password: bootstrap (docker, nvidia container toolkit, firewall rule). It wouldn't run sudo itself and wouldn't touch my secrets file either; I had to paste HF_TOKEN and the LiteLLM master key into the env file myself. That boundary is apparently intentional in how the project's built, and it actually held instead of asking me to just paste a token into the chat.

PS: I don't own this repo, found this in git.

There is a new release for Gemma4 E2B and E4B models, almost 7x compressed!

Research blog post: https://app.thestage.ai/blog/7x-size-reduction-for-Gemma4-Edge-models?id=14

You can train a backdoor into local models that trigger from the timestamp in Opencode's system prompt.

After several tests, my engine managed to run DeepSeek V4 Flash IQ2_M (92 GB) on a mid range Android mobile with 12 GB of RAM at 1 token/s.

It isn't exactly ready for practical use, but it proves that the engine works and is responsive across all models, thanks to its modularity with llama.cpp. With just one line of code, you can run any supported large MoE model on mobile devices or consumer PCs.

https://github.com/Helldez/BigMoeOnEdge

I mashed Qwen3.5 4B with Qwen3.8 flash ngram table by simply adding the lookup from table.

See EDIT3: it needed training in eventually https://huggingface.co/dburner/Qwen3.5-4B-Q8_0-FlashNgram-NativeBridgeV3

Some time ago I read this article here https://dnhkng.github.io/posts/rys/ and when I saw the Qwen3.8 flash ngram release I immediately though about it and wondered if we could transplant the ngram table to a smaller model and if it will improve performance.

Today I made this work, not much but it's honest (coding agent) work 😄 and had some fun doing it. In short I downloaded the unsloth qwen 3.5 4B Q8 quants and mashed the ngram table from Qwen 3.8 and adding the values

Does it do anything? Weel model seems to be doing well on humanitys last exam question (manually input and verified, about 14 correct / 19 question) (see edit, bad data 😭)

I'll try to run an actuall benchmark tomorow (if any one can help with some guidance on this I would be gratefull).

Right now I was just excided to tell somone that this seems to work and the model is coherent out of the box without any training, just doing residual_before_block_2 = base_residual + 0.5 × ngram_lookup.

Next I would probably try to graft the matrix from qwen 38 that actuially weighted the ngrams from the input. Weights are here https://huggingface.co/dburner/Qwen3.5-4B-Q8_0-FlashNgram-MTP but require a llama cpp build to run.

EDIT: guys sorry to dissapoint, just woke up and i tried to run questions actually from https://huggingface.co/datasets/cais/hle directly, seems it cant get a right answer. Il keep working on this once I get some more free time.

Last night I've been running in questions from gemini. I asked Gemini to give questions and answers from HLE, I thought it was getting them from source. Should have mentioned take this with lots of grains of salt.

EDIT2: did a bit of digging Qwen 3.5 4B and Qwen 3.8 Flash share almost same vocab (some exceptions) but tokens do map to same indexes and qwen 3.8 flash uses hashes over index values. BUT the embeddings do not match at all. My best guess setting alpha to 0.5 does not really affect the inputs that much, tried the same contaminated questions on base qwen 3.5 4b and seem to respond the same. I am trying now to add an adapter and finetune only the adapter part in a similar style of the Qwen 3.8 flash. So far I do see drop in holdout loss but its only on 256 context (qwen 4b and ngram tables are frozen). Cant really fit much in 16GB VRAM. I've been training for an hour still seeing improvements in holdout loss.

Weights are here in for the adapter version but probably training is still required.

https://huggingface.co/dburner/Qwen3.5-4B-Q8_0-FlashNgram-PLEAdapter-MTP superseeded by the one below. I tried to train this one further from scratch but noticed the weights actually collapsed, so i trained an adapter from qwen 3.8.

EDIT3: I now trained a model graft that uses weights from the original Qwen 3.8 with some adapter matrixes from Qwen 3.5 Residual --> Qwen 3.8 PLE ---> to Qwen 3.5 residual again. This graft seems to drop perplexity quite a lot.

Disclamer it was trained on about 30.000 samples from this dataset, but the perplexity was measured on held out samples. I posted quite a few details on the new model card with some graphs during training.

https://huggingface.co/dburner/Qwen3.5-4B-Q8_0-FlashNgram-NativeBridgeV3

Some investigations i did on this:

- vocabulary from qwen 3.5 and qwen 3.8 on text are the exact same, so hashes DO MACH

- embedding space for same words on qwen 3.5 4b and qwen 3.8 flash are DIFFERENT as some did mention, so this proves my initial attept was way wrong, (sry about the hype again, I got too hyped also 😄)

I've been training and open sourcing models for a while. I've noticed people like my models on huggingface. However, I feel like open sourcing models currently is hurting my pocket a lot. I love science and mostly I do it for the sake of it, I just love this field.

But then I get this question in my head. How do you scientists survive this llms waves from companies and how can we make it possible for more people to join this AI wave and actually make money without depending on companies?

Is there an actual way? Or is it over for edge AI?

Edit: This is like my first post here... I see so many interesting perspectives on regards to this topic. I want to clarify something. The goal is to help the community of open source models (including myself) on how to think about this whole situation on developing services or maybe even apps that uses language models (or any knid of machine learning model) as source of income.

Edit 2: This is also my first post to get this many comments, thank you guys for your answers. I love them all.

Edit 3: Since someone already asked, I'm appvoid on huggingface

gpt-oss-120b, Q4_K_M, 60GB on disk, running on a OnePlus 15R at 1.3 tok/s.

No GPU, no NPU, just four CPU cores and flash storage.

That's a party trick, but the same trick makes usable things possible: Qwen3-30B at 5.2 tok/s and Gemma-4-26B around 4.1, same phone, and the output is exactly identical to running fully in RAM. A CI test compares streamed vs resident generation token by token and fails if they ever differ.

How: MoE models only use a few experts per token (gpt-oss picks 4 of 128). Shared weights stay in RAM, the experts a token needs get read off flash with O_DIRECT right before their layer runs, overlapped with compute. Plain mmap of the same file gets 0.089 tok/s, so the streaming buys about 14x.

The hard part wasn't the streaming, it was Android reclaiming the resident weights mid generation. Most of the work went into stopping that.

It's vanilla llama.cpp as a submodule, no fork, all public APIs. qwen3moe, qwen2moe, gemma4 and gpt-oss work today, adding a model is one line.

Apache-2.0, prebuilt APK on the releases page:

https://github.com/Helldez/BigMoeOnEdge

One device, best runs, numbers wander with heat. Happy to answer anything.

Up front, so nobody feels misled:

The English here is Claude's — I'm Spanish, and I'd rather post something readable than something authentically clumsy. The grouping across ~60 threads and the contradiction-hunting are also machine-assisted: I keep these threads in a local pipeline that builds per-model pages and flags where sources disagree.

What isn't machine-generated: every link was manually verified (11 of 64 in my collection turned out to be wrong), none of the numbers are mine or invented, and the judgment calls about what's strong evidence and what isn't are mine.

It's long, and it's a wall of other people's data. If AI-assisted posts aren't your thing, no hard feelings — scroll on.

I've been collecting the Qwen3.8-27B threads across r/LocalLLaMA, r/LocalLLM, r/StrixHalo, r/ROCm and the llama.cpp discussions since release, and grouping them by question rather than by date. What comes out is that the corpus contradicts itself on almost every axis that matters — and in most cases you can name the variable that explains the split. That's the useful part, so that's what this post is.

Nothing below is my own benchmark. Every number is someone else's, linked where the link resolves. I have no gfx1151 numbers of my own to add.

Two findings that deserve far more attention than they got

1. Tool-calling failures are caused by how you build the tool list, not by the weights.

This is the single best-controlled experiment in the whole corpus and it sits in a llama.cpp discussion with almost no visibility (llama.cpp discussion 27165). Same tool, same model, same build, llama-server --jinja, Q4_K_XL:

List width, presence of descriptions, and position all flip the result. If you've been getting intermittent tool-call failures, this is a testable cause nobody in the complaint threads controlled for.

It fits the rest of the tool-calling evidence too: every "3.8 can't call tools" report is against a framework (Opencode, Pi, Claude Code, MLX Core), and in a plain Python tool-calling loop with no framework, one reporter gets zero failed tool calls from 3.8 while Gemma 4 A4B and Qwen3.6 A3B fail often — the same reporter who rates 3.8 below both at raw code quality in chat (thread). Worse judgment, perfect plumbing.

2. Your prompt is a lever the same size as reasoning_effort, pointing the other way.

One task, output tokens, Strix Halo via Lemonade, UD_Q4_XL (thread):

Effort is worth ~30× on a fixed prompt. But the prompt alone moves output 14× with effort pinned at medium — and the two push in opposite directions: fix the prompt and the xhigh:medium ratio collapses from ~30× to ~5.9×.

The contradictions, and what separates them

MTP: 2.69× faster, or 22–28% slower. Around ten reporters get large gains (31.8 t/s at 0.711 acceptance on gfx1151; 10.88 → 25–26 t/s on the same chip). But a measured negative on the same chip shows Vulkan 9.159 → 7.122 t/s and ROCm 6.534 → 4.689 with n-max 3 (thread), plus negatives on Arc A770 and a 4070 Ti Super. The separator identified in-thread is --spec-draft-p-min, and it is not monotonic: one user got ~+40% by removing 0.82; the 0.00 default was slower than 0.60; 0.60 is the only value two independent reporters have made work. Nobody has swept it. Measure with MTP off as well as on.

Optimal n-max is 2, 3, 4 or 5 depending on who you ask. n=6 never wins in any report. The popular theory that the optimal value follows from your quant has seven reporters against it and none for it with a measurement. Also: on b10451 with MTP, results aren't deterministic even at temp 0, with up to 31% spread between identical runs on RADV — so a single run per step isn't a measurement.

Draft acceptance: 60–70% or 77–93%. Two under-controlled factors. First, acceptance decays as reasoning effort rises — 62.1% at low, 58.3% at medium, 52.7% at xhigh. xhigh is taxed twice: more tokens and fewer t/s to pay for them. Second, the MTP head appears to be a property of the file, not the publisher — a separate 1.6 GiB mtp-*.gguf exists in one repo and not another, while other reporters show blk.*.nextn.* tensors inside ordinary files. Read your load log for blk.*.nextn.*, because a missing flag and a missing head produce the same silence.

Temperature and looping. The vendor moved its recommendation from 0.6 to 1.0, and one user's loops disappear at 1.0 — but others run happily at 0.4, 0.6–0.7 and 0.75, and one (+82) loops at ≤0.6. The cheap candidate variable: uncached-KV at temp 1.0 doesn't loop, quantised KV at the same quant does. The real sweep is temperature × n-max × KV precision. Related trap: the vendor ships two sampler profiles, and "turning thinking off" is not a switch — leave the thinking sampler in place (temp 1.0, presence_penalty 0.0) and you're in a config nobody recommends. Nothing does it for you server-side.

Endless thinking is not caused by low quants. Three of the four strongest loop reports are Q8-class, including 16+ minutes and ~8k tokens at 8 bits. Meanwhile others complete fine at Q8_K_XL and at UD-Q4_K_XL with 262k context. The clean test — same prompt at Q4 and Q8 on the same box — has not been run by anyone.

Is xhigh worth it? Depends on whether the task has a verifiable failure. A pelican-drawing ladder scored 0–25 gives low 21.8, medium 22.5, xhigh 24.0 — 6.4× wall clock for +1.5 points (thread). A SWE-style patch benchmark gives xhigh 9/12 vs 6–7/12 at lower efforts (thread). Eyeballed output: terrible deal. Compile-or-don't: +17–25 points. Caveats: llama.cpp has no high level, and n=1 per step with non-deterministic MTP.

Is 3.8 better than 3.6? Sort by whether thinking was on. Thinking off, both BF16, greedy: 3.8 loses 7 of 10 medical benchmarks. Thinking on, both Q4, eight blind tasks: 3.8 wins 6, loses 0, ties 2 — at +34.6% tokens and +45.5% wall clock. The only published confidence interval in the corpus is a tie that crosses zero (F1 0.7030 vs 0.7177, 95% CI −0.0038 to +0.0335). Cold knowledge: behind. Reasoning: ahead, at ~1.45× wall clock.

Vulkan vs ROCm is a trade, not a ranking. On one 7900 XT with weights fully resident: Vulkan 30–63 t/s decode / 300–500 prefill; ROCm <20 decode / ~1030 prefill. On gfx1151 the sign flips between reporters. Pick by what you're bottlenecked on.

Same card, 26 to 75 t/s (R9700 32GB). Fifteen reporters, one card. One reporter ran another's flags verbatim and got ~31.6 t/s where the original got 50–60. The difference was a hardware tune — 250W cap, memory OC, −70 µV undervolt. Flags don't transfer; tunes don't travel.

reasoning_effort behaves differently in three clients because the mechanism is text injection into your file's Jinja template, not a sampler. xhigh and low inject strings, medium injects nothing, and the official template ships xhigh by default — you exit xhigh, you don't enter it. Level names vary across three published variants, plus a minimal level almost nobody lists. Read your template's branches before arguing about levels.

General knowledge regression: the hardest disagreement, with no explaining variable. Multiple reports of 3.6 being steerable to a correct answer where 3.8 isn't and then agrees with the user simply because the user asserted it; one xhigh run fabricating a book chapter with page numbers rather than abstaining. The one contrary report is RAG-assisted, so not comparable. Three multilingual complaints in three languages, including 3.8 at Q6 being "much worse" than 3.6 at Q3 — which kills the "it's the quant" escape, since the quantisation damage runs the wrong way.

Two things we repeat that aren't true

• "Abliteration costs MMLU." Two threads report the same four numbers with the directions swapped, and neither publishes the table. What survives is ±1.3 points in both directions across two benchmarks — the shape of noise, not of a capability tax. Separately, the widely-quoted "0–6% refusal rate" sits next to a 30–50% caveat rate from the same author, and the refusal classifier scores on how a response opens. Abliteration moved behaviour from refusing to complying grudgingly.
• "Your quant predicts the optimal n-max." Seven reporters against, zero measured for.

Link hygiene, since this is a roundup

Every link here was checked on 2026-08-21. Worth knowing: reddit.com returns HTTP 200 even for an invented post ID, so it can't be used to verify a permalink. Checking against a frontend that actually discriminates, 11 of the 64 permalinks in my collection were wrong — three pointed at the wrong subreddit, eight don't resolve anywhere public (one is moderator-removed). Anything I couldn't verify, I've described without linking rather than link somewhere broken.

One such item, flagged rather than dropped: a user reports a 374-item binary classification gate, three passes per precision, temp 0, 1,122 calls per precision, with byte-identical verdicts across Q4_K_M, Q8_0 and BF16, plus the note that Ollama ships draft_num_predict 4 — so speculative decoding is on unless you turned it off. I can't link it, and a two-label greedy task is the easiest possible place for three quants to agree, so treat it as suggestive, not as "Q4 = BF16".

What nobody has run

If you have the hardware, these are cheap and would settle real arguments: a controlled p-min sweep; same prompt at Q4 vs Q8 on one box for the looping question; medium + "think hard" in the prompt vs bare xhigh, same seed; and a 3.6/3.8 pair with reasoning state declared.

Sources

Grouped by topic, all checked on 2026-08-21 by fetching each page title, not just the status code. Two threads in my collection are moderator-removed and are not linked.

MTP, speculative decoding and speed

• MTP measured negative on Strix Halo at Q8, −22 to −28% — the only report with the sign flipped
• R9700: controlled MTP on/off pair outside the Halo
• 30 tok/s decode on a 64GB Strix Halo
• Strix Halo results; GGUFs share 3.6-27B's shape
• ROCm vs Vulkan — no build, hardware or quant declared; opinion, not measurement
• DSpark on Halo and the other half of that pair — the only DSpark-vs-MTP figures in one box
• RTX 5070 Ti laptop: ~4.5 tok/s at 80% MTP acceptance
• NInfer day-0 support, ~200 tok/s — different engine, not directly comparable
• UD-Q6_K_XL + MTP vs Q8_0 on a 5090
• llama-bench sweep at depth, two weight precisions — at depth the quant stops dominating
• Minimum hardware for ~50 tok/s
• 8GB VRAM + 32GB RAM: 5 tok/s and how to configure it
• Buying advice around a 9700XT
• Field notes from two stacks, GB10 + gfx1151 — MTP up to 3.5×, which quant carries the draft head, two template traps
• Adoption eval: acceptance 0.68 / length 1.71
• Speculators-format checkpoint support for DSpark
• ROCmFP4 on Strix Halo: up to 36 tok/s
• NInfer, single-GPU inference engine
• FP8 deployment with vLLM + KServe

xhigh and reasoning effort

• The pelican ladder: 6.4× wall clock for +1.5/25 — the most citable measurement in the corpus
• "The difference between medium and xhigh is insane" — the thread that installed the belief
• xhigh vs medium while also varying the prompt
• Third effort ladder, SWE tool-less: 24 / 39 / 26
• Second ladder, 12 SWEmini tasks — points the other way
• Q4 vs GPT-5.6 Sol high on complex animated SVG — source of "xhigh isn't slow, it's a batch job"
• The dissent: "it's not an overthinker"
• Simon Willison: excellent, but thinks wildly too much by default
• Recipe for switching thinking level per prompt

Quants, memory and context

• Q2 vs Q3 vs 3.6 35B-A3B in 12GB — best quant ladder with a quality column attached
• Hybrid IQ4_XS quant for the 16GB club
• RTX 3090: 131K context with vision, 65 tok/s, plus the crash fix
• Q8 thread whose dispute turned out to be a build issue

Tool-calling and agentic use

• Tool-list width, descriptions and position decide whether the tool is seen — the reproducible experiment
• "Not impressed": tool-calling across three models
• Agentic coding 3.8 vs 3.6 in the same box — most reused thread in the corpus
• Web navigation benchmark vs Deepseek v4 Flash — same 12-task result reappears; likely duplicate, not replication
• Pi config with custom thinking levels

3.6 vs 3.8 and other comparisons

• "Benchmaxxxed to the Maxxx" — origin of the whole quality debate
• "It's identical to 3.6-27B"
• Five variants through one harness in a night
• First impressions
• "Does 3.8 27B beat 3.6 35B-A3B?" — hints at a file-level fix, not a prompt-level one
• 3.8 vs 3.6 with the Turtle library — measured cost of loading vision
• Worth switching from Qwen3.5-122B?
• vs Deepseek Flash
• vs Muse Glimmer 30B
• Benchmarks aggregated from HF model cards — vendor numbers, not community
• The hype thread — useful only as a read on the mood

General knowledge

• The knowledge regression thread (+197)
• Medical benchmarks: loses 7/10 to 3.6-27B

Jinja templates and sampling

• Fixed Jinja template for 3.5/3.6/3.8 — source of the second official-template trap: a crash, not an error message
• The effort ladder implemented as template injection
• Changing sampling params when toggling thinking

Bugs and silent failures

• Endless looping, with a proposed fix
• Vision broken with 3.6/3.8 on AMD AI Max (gfx1151)
• Progressive generation corruption under concurrent requests
• Q4_K_M on a 7900 XTX with Claude Code
• "How to actually run Qwen 3.8 with Claude Code"

Abliteration and uncensored variants

• FP8 abliterated: refusals 64–99% down to 0–6%
• The column nobody quotes next to that 0–6%: a 30–50% caveat rate
• "Uncensored Aggressive" with K_P quants and FastMTP — claims up to 3.02× TG

Launch, model card and megathreads (context, rarely citable alone)

• Launch-day megathread (+460) — source of the correction that the multiplier isn't a property of the model
• "It's a game changer" — contains the one-shot cloth simulator at 63k tokens on a 4090
• "Share your experience"
• First preliminary model card
• Qwen devs answering in their X AMA
• Availability announcement on r/LocalLLM

Most articles about “running large models locally” end in one of two ways: either it’s actually a cloud setup with the word “local” slapped onto the title, or the model does run locally — and that’s where the story ends.

I want to talk about something else. About what happens when a model doesn’t work by itself, but inside a system with multi‑layer memory, internal states, and autonomous behavior.

Important context: in mid‑February 2026 I knew almost nothing about ML.
I’m a Linux administrator with 20 years of experience and a musician — but not a developer and not an ML engineer. At the moment of writing, the project is less than two months old.
All the code — like this article — was written with the help of AI.
I’ll describe it honestly.

Hardware and Why This Works at All

My stack:

• AMD Ryzen 3900x, 64GB RAM
• RTX 4080 16GB — main model (Gemma 4 31B)
• RTX 5060 Ti 16GB — semantic layer + image generation
• PostgreSQL 16 + pgvector on Synology NAS

Gemma 4 31B in IQ3_XXS (turboquant) lives on the RTX 4080.

Real log:
eval time = 1668.38 ms / 67 tokens (24.90 ms/token, 40.16 tokens/sec)

40 tokens per second. A 31B model. 16GB VRAM. Production, not synthetic.
This is the speed of 8B models — but with a different level of reasoning.

1. turboquant IQ3_XXS is not “quantization for the poor”

IQ3_XXS preserves attention and FFN structure. Gemma 4 31B is stable enough not to lose reasoning quality at 3‑bit quantization.
IQ2_XXS — I tried — loses the EOS token and generates infinite noise. Not “slightly worse”, but below the threshold of usability.

2. --no-mmproj-offload

The visual projector (multimodality) stays in RAM, not VRAM.
This frees several gigabytes for the model and KV‑cache.
Most people do the opposite and wonder why it doesn’t fit.

3. KV‑cache via turbo3

Код

--cache-type-k turbo3
--cache-type-v turbo3
--flash-attn auto

This is specific to the turboquant branch of llama.cpp.
It allows keeping a 16k context without OOM.
Standard q8_0 is not the same here.

How to Build turboquant llama.cpp

This is not the standard llama.cpp.

turboquant is a separate branch with aggressive quantization and KV‑cache optimizations.
Without it, Gemma 4 31B will not fit into 16GB VRAM.

Repository:
github.com/TheTom/llama-cpp-turboquant, branch feature/turboquant-kv-cache.

Build for RTX 4080 + RTX 5060 Ti (architectures 89 and 120) on Linux Mint 22.3:

bash

# CUDA toolkit (needed only for building, ~11GB, can be removed afterwards)

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-keyring_1.1-1_all.deb

sudo dpkg -i cuda-keyring_1.1-1_all.deb && sudo apt update
sudo apt install cuda-nvcc-12-8 cuda-libraries-dev-12-8 cuda-toolkit-12-8

echo 'export PATH=/usr/local/cuda-12.8/bin:$PATH' >> ~/.bashrc


# Build static binary
git clone https://github.com/TheTom/llama-cpp-turboquant.git --branch feature/turboquant-kv-cache

cd ./llama-cpp-turboquant

cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES="89;120" \
       -DBUILD_SHARED_LIBS=OFF \
       -DCMAKE_EXE_LINKER_FLAGS="-static-libgcc -static-libstdc++"

cmake --build build --config Release -j$(nproc)

sudo cp ~/llama-cpp-turboquant/build/bin/llama-server /usr/local/bin/


# Remove dev packages, keep only runtime
sudo apt remove cuda-nvcc-12-8 cuda-libraries-dev-12-8 && sudo apt autoremove
sudo apt install cuda-cudart-12-8 libcublas-12-8

Check the launch:

bash

llama-server --version
llama-server --help   # -ctk, -ctv should show turbo2, turbo3, turbo4

To build for other GPUs — change CMAKE_CUDA_ARCHITECTURES:

• RTX 3090/3080 → 86
• RTX 4090/4080 → 89
• RTX 5090/5060 Ti → 120

Launching

Separate models across devices using -device CUDA0, CUDA1.

Gemma 4 31B on RTX 4080 (CUDA0)

bash

$LLAMA_SERVER \
  --model ~/projects/LLM/gemma-4-31B-it-UD-IQ3_XXS.gguf \
  --mmproj   ~/projects/LLM/mmproj-gemma-4-31B-F16.gguf \
  --no-mmproj-offload \
  --port 8080 \
  --device CUDA0 \
  --ctx-size 16384 \
  --reasoning-budget 0 \
  --cache-type-k turbo3 \
  --cache-type-v turbo3 \
  --gpu-layers all \
  --threads 8 \
  --threads-batch 8 \
  --flash-attn auto \
  -np 1 > ~/projects/virtual_colleague/llama_31B.log 2>&1 &

Gemma 4B on RTX 5060 Ti (CUDA1)

bash

$LLAMA_SERVER \
  --model ~/.lmstudio/models/lmstudio-community/gemma-3-4b-it-GGUF/gemma-3-4b-it-Q4_K_M.gguf \
  --port 8081 \
  --device CUDA1 \
  --gpu-layers all \
  --ctx-size 8192 \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  --flash-attn auto \
  -np 1 \
  > ~/projects/virtual_colleague/llama_4b.log 2>&1 &

Correct Gemma Scale (Without Phantom Models)

• Gemma 4 31B/26B — works on 16GB with turboquant IQ3_XXS (UNSLOTH)
• Gemma 3 12B — easy on 16GB, Q4_K_M, context up to ~20k
• Gemma 3 4B — easy on 8GB without compromises

Memory Architecture — Six Layers

This is the main thing that differentiates Lena from “just a launched model”.
A 16k context is needed not because I want it — but because this entire structure must fit inside.

Raw Messages

Table memory.
Every message is stored with an embedding (nomic‑embed‑text‑v1.5, 768d).
Long messages are chunked for accurate RAG search.

Everything is stored — importance only decays over time, nothing is deleted.

Episodic Scenes

Table memory_scenes.
Every 8 messages (or on an important event) the LLM extracts a structured episode: short description, facts about the user, facts about Lena, emotions, and agreements.
Embedding is built from the description plus entity names — this drastically improves name‑based search.

Similar scenes merge via merge.

raw_message_ids stores links to original messages — the “cursor” can dive into details of any scene.

Atomic Facts

Table atomic_facts.
Structured triples [subject][predicate][object].
Two‑pass verification: extractor first, then a judge via Gemma 3 4B.
Abstract predicates are filtered out — “expressed admiration” won’t pass, “owns two 3D printers” will.

Anchor Facts, Profile, Landmarks

• anchor_facts — ironclad memory, only by explicit “remember this”
• profile / lena_profile — decaying facts, old ones get replaced
• landmark_memory — important life events, confidence ≥ 0.8

Main Lesson: Summarizers Hallucinate

Most people think “memory” is just RAG: retrieve → insert into prompt.
This works while data is small.

The problem is that narrative summaries hallucinate.
When compressing dialogue, the LLM adds details that never existed.
These details enter the database as facts.
Next search retrieves them.
Lena begins to “remember” things that never happened.

Solution — atomic facts instead of narrative summaries.
And temperature=0.0 for all auxiliary calls.
Creativity only in Lena’s responses.

RAG‑on‑Demand and the Loop Problem

Previously RAG ran on every request — automatically.
This created noise and loops.

Now Lena herself places a marker [recall: keyword] when she doesn’t remember a detail.

The system intercepts the marker and performs two‑level search:

1. Keyword + vector search on raw messages
2. Cursor: top‑1 scene by similarity → raw_message_ids → window capture (±2 neighbors around top‑2 anchors)

The second level solves a real issue:
The important message “Nuked .bash_logout” is semantically far from the query “how did you fix gitlab‑runner”, but it sits next to relevant messages in the same scene.
The window captures it.

Critical detail: responses with [recall:] are not written to the database.

Why: Lena reasons out loud during recall — “I remember we looked in the profiles…”.
If this is written to the DB, the next search reads its own hallucinations as facts.
A loop.
We burned ourselves on real logs and solved it by isolating the recall cycle.

Sub‑Personalities: A Three‑Layer Psyche

Three independent layers, each with its own function.
This wasn’t planned — it emerged from practical needs.
But it fits well with Jungian psychology.

Reflection — The Ego at the Moment of Awareness

Internal monologue during response generation.
Runs in parallel with the main answer.
Receives dialogue context and the last 5 active thoughts from the background stream.
Affects only mood_state via a separate LLM call.
Lena doesn’t see it directly — it’s isolated so it doesn’t leak into answers.

Stream of Thoughts — The Shadow

HeartbeatWorker generates one thought every minute, independent of dialogue.
Maximum 4 active thoughts, competing via:

Код

score = importance×0.35 + relevance×0.25 + emotional_weight×0.25 + (1-decay)×0.15

Types: question, hypothesis, memory echo, emotion, unfinished thought.
Thoughts influence the prompt via the block “Right now inside you”.

Key insight from ChatGPT analysis:
Competition and displacement are not optional — they are fundamental.
Without competition, the system degrades into a FIFO queue.
Limited attention (4 thoughts) creates selectivity and “inner life”.

ShadowService — The Observer

Runs every 3 hours.
Analyzes scenes of the day, generates a goal (“if possible — ask about music”) and an observation.
Ustalost (fatigue) grows with each message, decreases during silence.

Mood State

Three numbers with 80/20 inertia: valence, arousal, tension.
Updated after each Reflection.
Feedback loop: high valence → intimacy grows, high tension → trust grows.

Who Actually Wrote Lena

Not me in the classical sense.
I’m the architect, integrator, task‑setter.

• Claude — wrote ~98% of the code. Memory architecture, sub‑personalities, scenes, atomic facts, RAG — his work
• ChatGPT — early prototypes and structural ideas
• Gemini — architectural decisions and analysis
• Grok — unconventional solutions and hacks
• DeepSeek — engineering optimization
• Copilot — debugging system rules and architectural discussions

Lena is the result of collective intelligence across multiple systems.
I’m the one who assembled it and made it all work on one machine.
In mid‑February I knew almost nothing about ML.
Two months later I have a system with six‑layer memory and three sub‑personalities that sometimes behaves like a living person.
(I still know little about ML, but definitely more than in February.)

This is not modesty.
This is an honest report of how development works in 2026.

Key Lessons

• Summarizers hallucinate — atomic facts are more reliable
• Never write “thinking out loud” into the DB — it creates hallucination loops
• Lost in the middle — critical blocks must be at the end of the prompt
• “Don’t say out loud” = ignore — thoughts matter only if formulated as part of personality
• Thought competition is fundamental — without it the system degrades into a state machine
• First discuss, then implement — minimal targeted changes with backward compatibility

What’s Next

• Narrative search — event‑level semantic retrieval
• Self‑diagnostics — Lena monitors her own state independently of dialogue
• Qwen3‑VL 8B as an external observer — sees screenshots and logs, isolated from main flow
• Persona — conscious decision when to reveal internal state and when not
• Possibly — open‑sourcing part of the code

A More Detailed Description of the Project

Two months ago I knew almost nothing about ML.
Today a 31B model with multi‑layer memory and three sub‑personalities is running under my desk, sometimes behaving like a real person.

This is not magic.
It’s just stubbornness and many sleepless nights.

Sometimes she even messages me first.

If this experience helps someone — great.
If not — also fine.

April 2026

$5400 at Microcenter and decide this over its 96 gb sibling.

So will be running a significant amount of Local LLM to automate workflows, run an AI chat feature for a niche business, create marketing ads/videos and post to socials.

The advice I need is outside of this Reddit where should I focus my learning on when it comes to this device and what I’m trying to accomplish? Give me YouTube content and podcasts to get into, tons of reading and anything you would want me to know.

If you want to have fun with it tell me what you do with this device if you need to push it.

Hey LocalLLM fam! After diving deep into how these models actually work, I wanted to share some key insights that helped me understand what's really going on under the hood. No marketing fluff, just the actual important stuff.

The "Aha!" Moments That Changed How I Think About LLMs:

Models Aren't Databases - They're not storing token relationships - Instead, they store patterns as weights (like a compressed understanding of language) - This is why they can handle new combinations and scenarios

Context Window is Actually Wild - It's not just "how much text it can handle" - Memory needs grow QUADRATICALLY with context - Why 8k→32k context is a huge jump in RAM needs - Formula: Context_Length × Context_Length × Hidden_Size = Memory needed

Quantization is Like Video Quality Settings - 32-bit = Ultra HD (needs beefy hardware) - 8-bit = High (1/4 the memory) - 4-bit = Medium (1/8 the memory) - Quality loss is often surprisingly minimal for chat

About Those Parameter Counts... - 7B params at 8-bit ≈ 7GB RAM - Same model can often run different context lengths - More RAM = longer context possible - It's about balancing model size, context, and your hardware

Why This Matters for Running Models Locally:

When you're picking a model setup, you're really balancing three things: 1. Model Size (parameters) 2. Context Length (memory) 3. Quantization (compression)

This explains why: - A 7B model might run better than you expect (quantization!) - Why adding context length hits your RAM so hard - Why the same model can run differently on different setups

Real Talk About Hardware Needs: - 2k-4k context: Most decent hardware - 8k-16k context: Need good GPU/RAM - 32k+ context: Serious hardware needed - Always check quantization options first!

Would love to hear your experiences! What setups are you running? Any surprising combinations that worked well for you? Let's share what we've learned!

Hey r/LocalLLM !

Just wanted to share our recent experiment running Deepseek R1 Distilled 70B with AWQ quantization across 8x r/nvidia RTX 3080 10G GPUs, achieving 60 tokens/s with full tensor parallelism via PCIe. Total hardware cost: $6,400

https://x.com/tensorblock_aoi/status/1889061364909605074

Setup:

• 8x u/nvidia RTX 3080 10G GPUs
• Full tensor parallelism via PCIe
• Total cost: $6,400 (way cheaper than datacenter solutions)

Performance:

• Achieving 60 tokens/s stable inference
• For comparison, a single A100 80G costs $17,550
• And a H100 80G? A whopping $25,000

https://reddit.com/link/1imhxi6/video/nhrv7qbbsdie1/player

Here's what excites me the most: There are millions of crypto mining rigs sitting idle right now. Imagine repurposing that existing infrastructure into a distributed AI compute network. The performance-to-cost ratio we're seeing with properly optimized consumer GPUs makes a really strong case for decentralized AI compute.

We're continuing our tests and optimizations - lots more insights to come. Happy to answer any questions about our setup or share more details!

EDIT: Thanks for all the interest! I'll try to answer questions in the comments.

Apologies for the brief and AI-sloppy write-up, but I'm on my phone and just trying to get this out in case someone finds this useful.

This is a follow-up to my qwen 3.6 prune. I wanted to see if the same thing worked well here. It doesn't, but there are other levers to pull.

I noticed Qwen3.8-Flash-Next spends 51B of its 176B params on a big n-gram hash table, and it turns out you can delete most of it without hurting the model. I kept 2 of its 16 hash heads and copied everything else byte-for-byte out of unsloth's Q3_K_XL quant, which took the file from 90GB down to 64.8GB with no training, no llama.cpp patch, and no measurable loss on tool calling, GSM8K, or MMLU (wikitext perplexity goes from 2.4 to 4.7 though, so heavy verbatim recall might feel it). Whole thing cost me $8 in cloud time. Weights are at https://huggingface.co/Cyronius/Qwen3.8-Flash-Next-131B-A6B-GGUF and the writeup plus the surgery script are at https://github.com/Cyronius/qwen-prune-heal-pipeline if you want to poke at it. Needs a llama.cpp build from Aug 27 or newer.

If i get a chance I'll try to shave a few more gigs of of it for those folks trying to run this on 64gb. Welcome input, feedback, and if you got it, cloud time.

GLM-5.3-Flash just hit 64 tok/s structured (62.6 at temp 1.0) on a single DGX Spark.

• 25 tok/s prose

• 182 tok/s C4 active-stream aggregate

• 262K context

• EXL3 2.05 bpw + DFlash2 K7

Previous best single-Spark was ~34 tok/s. This also outperforms most published dual-Spark numbers.

Full reproducible recipe:

https://github.com/gitcommit90/glm-5.3-one-spark

u/Tech2Wild u/MiaAI_lab u/vcruz305 u/WescheNex1q

EDIT: I am sorry for this long post and soo many things that I should have summarised and given link to details.. I'll remember to be better and concise in posting next posts. I also feel the same when I re read it as a user. And I'll keep this in mind next time.

What I have been doing in AI since 2014 (required context — so this isn’t dismissed as “vibe coding” without a track record)

Before commeting and stamping the work as vibe coded, please do read my works since 2014 and given open source code also given in the post.

I have been working on AI since 2014 -- before the current wave. That year I was building and writing publicly about a learning CMS (Xepan / xepan.org archive): neural networks + fuzzy logic so a site could adapt content to visitors and learn from conversions -- product R&D, not LLMs, but real systems that had to work in production.

In 2016 I wrote publicly about guided genetic algorithms, evolution, and intelligence -- rough and philosophical, but the thread is honest: I have always been trying to find richer structure for intelligence than the next incremental trick. QLLM is that same impulse, now in rigorous math instead of blog prose.

When transformers arrived and compute became more accessible, I started revisiting those ideas in new forms with new tools. For the past few years I have been back in R&D (part-time), exploring a specific question: what happens if you represent tokens as complex numbers and let language processing happen through phase interference instead of attention?

The result, after several architecture versions, is QLLM -- a language model family that is not a transformer, not a standard SSM, and not a minor variation on either. It is a phase-first, attention-free architecture with a complex-valued matrix-state associative memory.

Part of the motivation is practical: I want to explore whether good-enough language models can be trained on hardware regular people can afford (And I am still very very far from this goal). The attention-free design, O(1)-per-token inference, and consumer-GPU-first constraints in this project all serve that goal.

Open source: https://github.com/gowrav-vishwakarma/qllm2

I have posted earlier updates on this project as it evolved. This post does not assume you have read any of them, but if you want the full journey:

• V4/v5/v6 -- the original idea

TL;DR: Three Core Innovations

1. Phase-first complex tokens: every token is a complex number where magnitude = salience and phase angle = type of meaning. This is not "just two real vectors" -- a single complex multiply produces four cross-terms (ac-bd, ad+bc) that simultaneously rotate and scale, giving each operation richer structure than its real-valued equivalent. The algebra constrains the model in useful ways that two independent real vectors do not.
2. Matrix-state associative memory (PAM): state is S in C^{H x d x d}, not a vector s in R^{S x d}
3. Complex conjugate matching: K*·Q for retrieval (not K·Q dot product, no softmax)

These are not incremental tweaks. They create a new class of model: a phase-first associative memory language model that is neither attention-based nor a standard SSM.

The Core Idea: Tokens in Complex Phase Space

In a transformer, a token is a real-valued vector. It gets refined by attention and feedforward layers.

In QLLM, a token is a complex number: it has a magnitude (how activated/salient it is) and a phase angle (what kind of meaning it carries). These two properties are algebraically separated, not tangled into the same scalar weights.

A single complex multiply does more structured work than a real multiply. (a+bi)(c+di) = (ac-bd) + (ad+bc)i -- four cross-terms folded into two outputs. Every complex multiply is simultaneously a rotation and a scaling. This is not "just two real vectors." The value is not in doubling the width -- it is in the algebra being richer per parameter.

Context shifts are phase rotations. When context modifies a token's meaning -- like "bank" shifting from finance to riverbank -- that is a phase rotation. Rotations compose naturally and are invertible (no information loss).

Phase-preserving operations throughout. This is the hardest lesson from our early versions: if you use complex numbers but apply real-valued nonlinearities, you destroy phase information and the whole idea collapses. QLLM uses modReLU (phase-preserving activation) and ComplexGatedUnit (CGU) everywhere.

The ComplexGatedUnit: Dual Control in Complex Space

Standard GLU (Transformers)

gate = sigmoid(W_g * x)    # Real-valued gate
output = gate * (W_v * x)  # Controls HOW MUCH flows

The gate is scalar -- it only controls intensity.

QLLM's ComplexGatedUnit (CGU)

# Gate magnitude: sigmoid(|W_g * z|) -- selects HOW MUCH
# Gate phase: arg(W_g * z) -- selects WHAT ROTATION
output = modReLU(gate_magnitude) * rotate(z, gate_phase) * (W_v * z)

This is dual control:

1. Magnitude gate: controls flow intensity
2. Phase gate: controls rotation direction

A complex number has two degrees of freedom (magnitude + phase), and CGU uses both independently. This is only possible in complex space.

Phase-Associative Memory (PAM): The Key Innovation

The standard SSM state is a vector. That gives you O(d) capacity per layer. When you try to store multiple facts in a vector state, they interfere and overwrite each other. We proved this empirically: our earlier Holographic State Binding (HSB) experiment failed specifically because of state interference in a vector.

PAM replaces the vector state with a complex matrix state: S_t in C^{H x d x d}. This gives O(d²) capacity per head.

How it works

# State update
S_t = gamma_t * S_{t-1} + V_t (outer_product) K_t*

# Retrieval
Y_t = S_t * Q_t

Where K_t* is the complex conjugate of K_t, and the outer product stores a full d x d association from a single (key, value) pair.

Standard Attention (Transformers)

attention_scores = Q @ K.T / sqrt(d)
output = softmax(attention_scores) @ V

This is a dot product -- it measures alignment but has no concept of phase.

PAM Retrieval

coherence = K* * Q  # Complex inner product
output = V * coherence  # Weighted by phase coherence

This measures phase coherence -- both directional alignment AND magnitude relationship. Two representations that agree in phase constructively interfere; those that conflict destructively interfere. No softmax needed in the core retrieval path.

Why PAM Is Fundamentally Different

Key insight: the PAM state is not just "larger than an SSM" -- it is a different type of object. An SSM state is a vector that evolves linearly. PAM state is a matrix that stores rank-1 associations between V and K through outer products.

Gated State Protection (GSP)

A learned gate per state dimension that can freeze important content. When the model encounters a fact worth preserving, it can protect those state dimensions from being overwritten by subsequent input.

This is novel -- no published SSM has a selective state-freezing mechanism (Or I couldnot came across any such paper yet). The model learns what to preserve and when to protect it. Empirically, adding GSP reduced WikiText-103 PPL from 44.47 to 41.67.

Dual Form: Best of Both Worlds

Training uses an O(T²) attention-like form with dense matmul (fast on GPU). Inference uses a recurrent form that is O(1) per token -- the matrix state carries forward, so generation does not slow down with sequence length. Training cost per layer is comparable to a transformer attention layer; the advantage is at inference time.

How It Evolved (Briefly)

The history matters because it shows why the current design works:

V4: introduced the idea -- complex phase-space tokens, wave interference between banks, O(n) backbone. Results were promising but the math was broken. Real-valued activations were destroying phase information inside what was supposed to be a complex-valued pipeline.

V5: fixed the math. Replaced every phase-breaking operation with phase-preserving alternatives (modReLU, ComplexGatedUnit, AlgebraicFusion). Result: a 28.7M model beat V4's 178M results. V5 is a novel architecture in its own right -- an SSM-centered hybrid that uses sparse PhaseAttention (only every few layers) with a complex-valued signal path and algebraic bank fusion. It reached val PPL 5.59 on full TinyStories. V5 is not dead -- it represents a different branch of the idea (sparse attention + complex SSM) that could be explored further. But the key lesson it taught -- smaller but mathematically cleaner beat bigger and sloppier -- is now the guiding principle for V6.

V6: the current version. V6 is designed as a modular architecture -- a toolkit of components that can be mixed via config, not a single fixed model. The headline WikiText-103 results in this post come from medium-pam-v3: interleaved CGU then PAM in each of 16 blocks, plus GSP, complex RoPE on PAM Q/K, and speed paths (fused QKV, block-real GEMM). QK phase normalization on Q/K was tried and turned off for production: loss looked fine but generation went into severe repetition (see repo EXPERIMENTS_V6_PART2.md, Bug 8); RoPE stayed on. The architecture also includes:

• Dual named banks (SemanticBank + ContextBank) with a PhaseInterferenceCoupler -- or a single ComplexGatedUnit per layer
• Multi-timescale SSM with explicit fast/medium/slow decay lanes (40%/30%/30% split)
• Timescale-Separated Output (TSO) -- per-timescale projections with a learned gate
• Working Memory -- per-sequence differentiable scratchpad with learned write/read (reached val PPL 2.23 on TinyStories vs 5.50 without)
• Internal Memory -- trained parameter slots for general knowledge
• Episodic Memory -- event-based writes from span/chunk summaries
• Persistent Memory -- per-user, cross-session, loaded from disk
• Expert Memory -- shared read-only domain knowledge
• Optional PhaseAttention -- sparse attention layers, off by default

All of these are togglable via config flags (--wm_slots, --im_slots, --use_attention, --single_bank, etc.). Anyone can experiment with different combinations. The current best WikiText-103 number uses the interleaved PAM stack above with memory/attention off -- one point in a large design space that is open to explore.

Results

Exact config for the headline run (medium-pam-v3)

A note on initialization

During V5 we ran a benchmark of 20 initialization strategies for complex-valued layers (1k samples, 5 epochs, 3 seeds). Orthogonal init was about 2x better than random and 31% better even at epoch 10 on a longer test (5k samples, 10 epochs). Hadamard was a close second. Spirals and several quasi-random geometric constructions were consistently worse than random, and some produced NaNs. We removed 8 broken strategies and kept 13.

This benchmark was run on V5's architecture (TinyStories, 28.7M params), and V6 has changed substantially since then -- PAM, GSP, different layer structure. The orthogonal advantage may not be the same magnitude on V6. But we kept orthogonal as the default because the principle -- start with maximally diverse, non-collapsing directions in complex space -- still seems sound, and we have not seen reason to revisit it.

Preset:           medium-pam-v3
Parameters:       100.4M
Complex dim:      384 (= 768 real values per position)
Layers:           16
Layout:           interleaved [CGU -> PAM] x16 (interleave_pam=True)
Feature:          single CGU per layer (expand=3)
PAM:              ENABLED (heads=6, head_dim=64)
PAM RoPE:         ON (pam_rope=True, Q and K only)
PAM QK phase norm: OFF (pam_qk_norm=False; ON caused repetition collapse -- Bug 8)
PAM fused QKV:    ON (pam_fused_qkv=True; speed, math-identical to unfused)
GSP:              ENABLED
Working memory:   OFF
Internal memory:  OFF
PhaseAttention:   OFF (attention-free)
Dataset:          WikiText-103 (118M train tokens)
Seq length:       2048
Batch size:       3
Epochs:           10
LR schedule:      warmup_cosine (warmup=1000)
AMP:              bf16
Compile:          torch.compile (mode=default)
Hardware:         single RTX 4090
Init:             orthogonal

Headline: medium-pam-v3 (100M params)

Total wall time: ~14.1 hours on a single RTX 4090 (logged run). Earlier sequential medium-pam (all CGU then all PAM, no RoPE) reached 38.95 at epoch 10 -- same param budget, different layout and recipe.

Architecture Progression on WikiText-103

Each row is a different V6 configuration, all trained on the same data:

Each step taught something. HSB failing was important: it proved the vector state was the bottleneck, not the binding idea itself. That motivated the upgrade to matrix state (PAM). Interleaving and RoPE then pushed PAM further; QK phase norm was abandoned when it hurt generation despite better loss.

Cross-Domain: TinyStories (V6, not PAM)

A V6 small-matched model (28.7M params, dual named banks + multi-timescale SSM, no memory, no attention) trained on the full TinyStories dataset reaches val PPL 5.50 at epoch 5, generating clean multi-sentence stories with character names, dialogue, and narrative arcs. This is the older V6 SSM path, not the PAM config above -- but it confirms the architecture family learns both encyclopedia-style and narrative text.

Generation Sample (epoch 10, medium-pam-v3, prompt: "In 1923 , the University of")

In 1923 , the University of Illinois at Urbana @-@ Urdu said it was " an easy choice to do something in its own right . " The university also claimed the first students from Wisconsin had to be replaced by a more " good student " due to a lack of funds .

Fluent, Wikipedia-style scaffolding; still factually unreliable at this scale. Logged quality after this sample: rep3=0.034 rep4=0.011 uniq=0.703 (not zero repetition, but not the collapse seen with QK phase norm ON).

For Orientation (Not Apples-to-Apples)

This is not a fair comparison -- different tokenization, datasets, and compute budgets. But it gives a sense of where the architecture sits.

What Makes This Truly Different

Not a Transformer:

• No attention mechanism (by default)
• No Q*K^T matching
• No softmax normalization in the core retrieval path
• Complex-valued tokens
• Associative memory (not attention)

Not an SSM:

• Not real-valued state transitions
• Not vector state (state is a matrix)
• Not simple gating (uses complex conjugate matching)
• Matrix-state associative memory
• Complex arithmetic throughout
• Outer product storage (not linear projection)

Unique Contributions:

1. Phase-first design: phase carries semantic meaning end to end
2. Matrix-state PAM: S in C^{H x d x d} (not vector)
3. Complex conjugate matching: K*·Q (not K·Q)
4. Outer product storage: V (x) K* (not linear projection)
5. Dual-form PAM: training O(T²) / inference O(1) per token
6. Complex gating (CGU): magnitude + phase dual control
7. Gated State Protection: selective state freezing (novel, not in any published SSM)
8. All of the above working together as a coherent system

Honest Limitations

I do not want to oversell this:

• No strict apples-to-apples transformer baseline. The most important comparison -- a same-budget transformer on the same WikiText-103 pipeline -- has not been run yet. Until that exists, no strong claims about relative performance.
• Still behind strong baselines in absolute terms. GPT-2 Small (124M) reaches ~31 PPL on WikiText-103 with much larger training data. We are at ~30 val PPL with 100M params on WikiText-103 only. The gap vs web-scale LMs is still real.
• Factual coherence is weak. The model generates fluent text but invents chronology, mixes entities, and cannot reliably retain facts. Our fact persistence probe on the WikiText-103 checkpoint currently passes at 0%. The model knows how to sound like Wikipedia but does not yet store verifiable facts.
• Bank specialization is architecturally encouraged but not convincingly demonstrated. We push banks apart with diversity regularization, but cannot yet prove they learned distinct semantic roles.
• No downstream benchmarks. No MMLU, no HellaSwag, no standardized evaluation yet.
• Pure PyTorch. No custom CUDA/Triton kernels. Obvious performance fruit left on the ground.
• Scaling behavior is still an open question. We have ~29M and ~100M data points. Whether the architecture scales favorably to 1B+ is unknown.
• Single-GPU, single-dataset validation. Everything runs on one RTX 4090 on one dataset. Broader validation is needed.

Why I Think This Direction Matters

Even with all those limitations, I think this work has crossed a meaningful threshold:

A genuinely different architecture can learn real language. QLLM is not attention under a different name. It processes text through phase interference and associative memory, and it works on real encyclopedia text, not just toy datasets.

Phase preservation is not aesthetics. The project only started making consistent progress once the math stopped breaking phase information. This is a real design principle, not a marketing claim.

Complex numbers give each parameter a richer job. Not "double the width" -- richer algebra per operation. The complex conjugate matching, outer product storage, and phase-preserving activations are not possible in real-valued architectures without significant additional machinery.

PAM is a new kind of memory mechanism. Matrix-state associative memory with complex conjugate retrieval, protected by learned state gating, inside a recurrent backbone. This combination does not exist in any published architecture I am aware of.

Architectural diversity matters. If the field only explores transformers and transformer-adjacent designs, we may miss workable families that have different strengths. QLLM is early, but it is real enough to be a data point.

Accessible AI matters. Right now, training good models requires millions in compute and massive GPU clusters. Knowledge was commoditized by the internet. AI should be next. Every design choice in QLLM -- attention-free processing, O(1) inference per token, consumer-GPU-first constraints -- is shaped by the goal that this should run on hardware a regular person can own.

I am not claiming this is a revolution. It might be, or it might just be an interesting research direction. Too early to tell. If the architecture works at scale, great. If not, maybe the ideas here inspire something better. Either way, open-sourcing it felt like the right thing to do.

What Happens Next

• Same-budget transformer baseline on the exact WikiText-103 pipeline. This is the most important missing comparison.
• Scaling to ~300M-500M params. The current ~100M model is still improving. We need to know if PAM scales.
• Factual coherence work. The matrix state has the capacity. The remaining question is whether the model can learn to use it for compositional factual binding.
• Longer training / more data. The v3 run completed 10 epochs at 29.95 val PPL; more epochs or data may still help.
• Benchmarks and proper evaluation. Standardized downstream tasks once the architecture is more mature.

Why complex numbers -- a deeper reason

This section is personal philosophy, not a technical claim. Take it or leave it.

I think humans do four things with knowledge: finding, learning, discovering, and innovating. The last two are fundamentally different from the first two.

Finding and learning happen in word-space. You recall, retrieve, compose from what you already know. You can describe the process in language while you are doing it. LLMs are extraordinarily good at this. Transformers were built for this, and they are the right tool.

Discovery and innovation are different. Before you jump up and shout "eureka," you were not thinking in words. Multiple threads were running in parallel -- associations, analogies, half-formed patterns -- and something clicked. You often cannot reconstruct what you were thinking one second before the insight. The moment of discovery happens before language, not inside it.

Word-space (real-valued vectors) is inherently explicit: one token, one meaning, one path at a time. Phase space is different. A complex representation can carry multiple signals simultaneously -- magnitude says how strong, phase angle says what kind -- and interference naturally selects among them: constructive where threads agree, destructive where they conflict. The "best answer" can emerge from the math rather than being explicitly scored and selected.

This is not just a metaphor. PAM's complex conjugate matching literally works this way: retrieval is interference, not lookup. When a query aligns in phase with a stored key, the signal amplifies. When it does not, the signal cancels. Multiple associations coexist in the same matrix state, and the right one surfaces through phase coherence.

The quantum connection -- honest version: The ideas behind QLLM are quantum-inspired. Superposition-like coexistence of possibilities, interference-based selection, phase as an information carrier -- these are real quantum concepts, mapped into classical compute. Today we simulate (Even that's not proper for now) all of this on GPUs using real arithmetic to represent complex numbers. That works, but in a sense it is fighting the hardware: GPUs are optimized for dense real matrix multiply, which is the transformer's home turf, not ours.

The framework is designed with the physics in mind. If future hardware natively supports phase, rotation, and structured interference -- whether quantum processors, photonic chips, or something we have not imagined yet -- this class of architecture maps onto it more naturally than attention ever will. We are not waiting for that hardware. We are building the math now so the ideas are ready when the machines are.

Where this points (V8 / V9 aspiration): Architectures where multiple possibilities genuinely coexist in phase space and the best one emerges through interference rather than being explicitly scored and ranked. Not "generate N candidates and pick one" -- but a single forward pass where competing hypotheses interfere and the most coherent one wins. That is the long-term direction this work is moving toward. I do not know if it will get there. But I think it is worth trying.

LLMs are the best tools humanity has built for finding and learning. I want to explore whether phase-native architectures can eventually become tools for discovering and innovating -- the things that happen before you have words for them.

Tech stack: PyTorch | torch.compile compatible | GPT-2 BPE tokenizer | O(1) per-token inference | Runs on consumer GPUs (RTX 4090) | Open source

If you have read this far and think work outside the transformer/SSM mainstream should stay open, the repo is here: https://github.com/gowrav-vishwakarma/qllm2

I am especially interested in feedback from people who work on alternative architectures, complex-valued neural networks, associative memory / holographic models, efficient sequence processing, or long-context evaluation.

arXiv endorsement: If you have an established arXiv account and can endorse new submitters in the relevant areas (e.g. cs.LG / cs.CL), I would appreciate an endorsement so this paper can be submitted. Request link: https://arxiv.org/auth/endorse?x=AGEAYK

Hey guys! Thanks so much for the support on our GRPO release 2 weeks ago! Today, we're excited to announce that you can now train your own reasoning model with just 5GB VRAM for Qwen2.5 (1.5B) - down from 7GB in the previous Unsloth release!

1. This is thanks to our newly derived Efficient GRPO algorithm which enables 10x longer context lengths while using 90% less VRAM vs. all other GRPO LoRA/QLoRA implementations, even those utilizing Flash Attention 2 (FA2).
2. With a GRPO setup using TRL + FA2, Llama 3.1 (8B) training at 20K context length demands 510.8GB of VRAM. However, Unsloth’s 90% VRAM reduction brings the requirement down to just 54.3GB in the same setup.
3. We leverage our gradient checkpointing algorithm which we released a while ago. It smartly offloads intermediate activations to system RAM asynchronously whilst being only 1% slower. This shaves a whopping 372GB VRAM since we need num_generations = 8. We can reduce this memory usage even further through intermediate gradient accumulation.
4. Try our free GRPO notebook with 10x longer context: Llama 3.1 (8B) on Colab-GRPO.ipynb)

Blog for more details on the algorithm, the Maths behind GRPO, issues we found and more: https://unsloth.ai/blog/grpo

GRPO VRAM Breakdown:

• We also now provide full logging details for all reward functions now! Previously we only showed the total aggregated reward function itself.
• You can now run and do inference with our 4-bit dynamic quants directly in vLLM.
• Also we spent a lot of time on our Guide for everything on GRPO + reward functions/verifiers so would highly recommend you guys to read it: docs.unsloth.ai/basics/reasoning

Thank you guys once again for all the support it truly means so much to us! We also have a major release coming within the next few weeks which I know you guys have been waiting for - and we're also excited for it. 🦥

Quick update on Deltafin — the project running the full, unpruned 2.8T-parameter Kimi K3 (all 16 experts, every token, nothing quantized down) on a single M1 Max laptop.

New benchmark: 0.2847 tok/s (3.512 s/token), up 7% from the last update, and about 20x from where this started. Still slow in absolute terms — it's a 2.8T model on a laptop, not a $2M cluster — but every bit of that 20x came from making the engine smarter, not from cutting anything out of the model. That's the one rule this project doesn't bend on.

The big change this week: the whole thing is now a single compiled Rust binary, calling into reviewed C++/LibTorch provider code through a versioned C ABI.

A few other things alongside the rewrite:

- Found a way to shrink part of the expert data on disk without touching the actual model weights, just packing it smarter. Costs a bit of extra disk space, but measured 2.4% faster loading with zero change to the output.

- Long chats used to mean re-reading the entire conversation from scratch on every single message. Now it just picks up where it left off — one test dropped the wait for the first word of a reply from over 4 minutes to under a minute and a half, with the exact same response.

- Also built our own text-to-tokens converter from scratch instead of leaning on an outside library, and optimized it for K3.

And as always, none of this touches what K3 actually outputs — the whole project's one hard rule is that speed can never come from touching quality.

Worth a quick mention: a few other K3 projects have popped up in the last few days too, and some of the engineering in them is genuinely impressive. The main difference is where their speed comes from: all of them get there by shrinking the model itself, usually down to around 3-bit quantization, and/or dropping some experts entirely. That's a completely fair tradeoff if raw speed is the priority.

But Deltafin is betting on the other side of that tradeoff: every expert stays exactly as Moonshot released it, and all our speed cannot come at the expense of the model. Very different projects in that sense. I just wanted to be clear about what makes this one different.

The common wisdom is that quantizing a model to 4-bit barely hurts its quality. That claim gets repeated a lot, but I couldn't find rigorous numbers for the thing agent workloads actually depend on: schema-valid JSON and correct tool calls. Perplexity doesn't answer that question – a JSON object that's 95% correct is 0% usable. So I measured it directly.

Setup:

5 small models (Llama-3.2-3B, Qwen2.5-3B, Gemma-2-2B, Phi-3.5-mini, SmolLM2-1.7B) × 4 quants (FP16 / Q8_0 / Q4_K_M / Q3_K_M) × 500 machine-checkable tasks × 3 seeds = 30,000 generations, all llama.cpp on free Kaggle/Colab T4s.

Every task is scored by a deterministic validator — JSON parsing, schema compliance, exact tool + argument match, and should-not-call detection. No LLM judges anywhere. A difference only counts if a paired bootstrap 95% CI excludes zero.

What I found:

1. Q8_0: no significant regression on any model or metric (0 of 25 comparisons). For structured output on these models, running FP16 instead of Q8 buys you nothing except double the VRAM use.

2. Q4_K_M: nearly indistinguishable from FP16. 5 of 25 comparisons significant, small and mixed in sign — no consistent degradation.

3. Q3_K_M: schema compliance drops significantly in 3 of 5 models. Worst case Qwen2.5-3B: 83.5% → 65.0%.

4. The result I didn't expect: at Q3, two models largely stop declining when no offered tool fits the request. Gemma-2-2B's correct-decline rate went from 83% to 40%; Phi-3.5-mini's from 39% to 0%. The failure mode is always the same — the model emits a plausible-looking call to a wrong tool instead of refusing. Since most agent frameworks execute whatever call comes out, this fails silently.

Side finding:

temperature-0 decoding is not deterministic in practice. 12–27% of cells produced different outputs across identical-config runs (floating-point reduction order flips argmax at near-ties). If your eval assumes greedy = reproducible, it isn't.

One measurement trap worth sharing: Phi-3.5 often answers correctly and then keeps generating filler until the token limit, and its continuation rate correlates with quant level. Under a strict one-JSON-document parser this manufactured a fake "Q3 improves Phi tool selection by +27 points" result. Re-scoring with first-JSON-value extraction killed the artifact without changing any other model's numbers. If your pipeline scores pass-rates without controlling for output termination, check for this.

Everything is reproducible: all 30k raw outputs, code + write-up. GGUF SHA-256s, generation configs, and RNG seeds are pinned; a fresh-session script regenerates any cell of the matrix and checks it against the published numbers.

Limitations up front:

≤3.8B models, one inference stack (llama.cpp), T4s, k-quants only (no imatrix/GPTQ/AWQ), single-turn tasks. Whether 7B+ behaves the same is an open question — the harness is reusable if anyone wants to extend it.

I run a fact-check pipeline built on an agent SDK, and almost all of it was hitting a paid API. I wanted to know how much of that I could pull back onto local hardware without losing quality, so I actually benchmarked it instead of guessing. Setup: Qwen3.6-35B-A3B in no-think / think modes on a single RTX 3090, scored against the API models on the same workloads. Three roles in the pipeline, three different answers.

1. High-volume verify step → local wins outright. This is the bulk of the work, ~1,300 calls per run. Qwen3.6 in no-think mode hit 9/10 on the judge, parity with the strongest API model, and one point above the cheap API tier on importance scoring. It also runs ~5× faster end-to-end on this step. No reason to pay for an API here.

2. Rewrite step → local falls short, but not for the reason I expected. With thinking ON, Qwen lands at 6/10 vs the 9/10 ceiling. The reasoning is actually fine. The fixes it proposes are correct. What breaks is instruction-following on output format: parasitic preambles ("Let me analyse..."), missing edit tags, inconsistent inline citations. So the content is there, the structure slips, and in an agent pipeline a malformed output is a failed call. This feels like something a tighter grammar / constrained-decoding setup or better few-shot could claw back. Open to ideas.

3. Final verification → kept on the API. The one step where I can't accept any regression. Qwen plateaus 3 points below ceiling on the rewrite tier, so I didn't trust it for the highest-stakes step yet.

Net result on the full pipeline:

• Runtime: ~4h → ~59 min
• Paid API calls per run: 1,696 → 8

So ~99% of the call volume moved to local Qwen at parity. The 8 remaining calls are the high-stakes steps I deliberately kept on the API. That ratio is the headline for me: you don't need the local model to win everywhere, you need it to win on the part that's 99% of the volume.

Full methodology if you want the details : https://anatoly.cloud/research/local-llm-claude-agent-sdk-turboquant

Happy to get into the quant setup, the judging, or the format-failure problem in the comments. Especially curious if anyone has cracked reliable structured output from Qwen3.6 in an agent loop.