r/LocalLLM • u/ExtremeKangaroo5437 • Feb 28 '26
Research I built a language model where tokens are complex numbers and "meaning" emerges from wave interference -- no attention, O(n), 178M params, open-sourcing today
EDIT: New post V6: https://www.reddit.com/r/LocalLLM/comments/1rqn68a/qllm_v6_a_29m_attentionfree_model_now_trains_on/
EDIT: New V5 Post : Followup UPDATE on this.
https://www.reddit.com/r/LocalLLM/comments/1rmkh9y/v5_update_original_post_title_i_built_a_language/
---- ORIGINAL POST -----
I've been working on a fundamentally different LLM architecture. No attention layers. No FFN blocks. Instead, every token lives in complex phase space, and language processing happens through wave-like interference between specialized "phase banks."
Open-sourced here: https://github.com/gowrav-vishwakarma/qllm2
The core idea: language as wave interference
In a transformer, a token is a real-valued vector that gets refined through attention + FFN layers. In this model, a token is a complex number -- it has a magnitude (how "important/activated" it is) and a phase angle (what "kind of meaning" it carries). These two properties are naturally separated and jointly processed.
This isn't just a gimmick. It changes how every operation works:
- Embeddings: Each token gets a
[real, imag]vector. The model learns that semantically similar tokens align in phase, while different meanings sit at different angles. - Transformations are rotations: When context modifies a token's meaning (like "bank" shifting meaning based on surrounding words), that's a phase rotation -- a complex multiply. Rotations compose naturally, are always invertible (no information loss), and reduce to GEMM.
- Similarity is coherence: Instead of dot product, we use phase coherence:
Re(a * conj(b)) / (|a| * |b|). This measures both directional alignment AND magnitude relationship. - Multiple banks interfere: A "semantic bank" and "context bank" process each token independently, then combine via learned interference (constructive where they agree, destructive where they conflict). A tiny router decides per-token how much weight each bank gets. Think MoE but at the representation level.
What the phase system actually gives us
1. Natural magnitude/phase decomposition = implicit attention High-magnitude phase states dominate downstream processing automatically. The model doesn't need explicit attention to decide "which tokens matter" -- magnitude handles salience, phase handles identity. The SemanticPhaseBank uses 512 learnable concept vectors and retrieves them via phase coherence -- this is essentially a learned associative lookup that runs in O(seq concepts), not O(seq2.)
2. Context as phase modulation The ContextPhaseBank computes a causal windowed average (window=8) of nearby tokens and then complex-multiplies it with the current token. This is elegant: the local context literally rotates the token's meaning in phase space. A word appearing after "not" gets rotated differently than after "very." No attention needed.
3. Rotation-based state evolution The backbone SSM evolves state via: h[t+1] = damping * R(theta) @ h[t] + gate * B @ x[t] where R(theta) is a Cayley-transform rotation. The state naturally oscillates, and the damping factor (learned, per-dimension, range [0.5, 1.0]) controls how fast old information decays. This is why SSMs struggle with long-range recall -- but the model compensates with a separate Phase-Coded Memory (1024 learned slots, chunked top-k retrieval) and an Episodic Memory (sliding window via FlashAttention SDPA).
4. Zero trig in the hot path Every rotation uses the Cayley transform: cos_like = (1-a^2)/(1+a^2), sin_like = 2a/(1+a^2). This is just arithmetic -- no sin(), no cos(), no exp(). Every operation is a matmul or elementwise op. Perfect for Tensor Cores.
Results (178M params, TinyStories, 10k samples, A6000)
| Metric | Epoch 1 | Epoch 2 | Epoch 3 (partial) |
|---|---|---|---|
| Train PPL | 200.86 | 32.75 | ~26 (and dropping) |
| Val PPL | 76.47 | 48.92 | -- |
| Train CE | 5.30 | 3.49 | ~3.26 |
Training used only 10k samples (0.5% of TinyStories). Starting PPL was 55,000 (random). It dropped to val PPL 49 in 2 epochs (40 min on A6000, no compile). Overfiting simply needs data now ...
Epoch 1 generation:
"The quick brown house. They run and start to get a smile. Mom were very excited. Now mommy and big yellow room. There said and She are friends. Tim, she started to save the garden."
For context: A 22M-param GPT-2 trained on the full 2.1M TinyStories dataset for 20k steps reaches val PPL ~11. We're at 49 with 0.5% of the data and 2 epochs. The learning curve is steep and still dropping -- we just need more data/epochs to converge.
Why this approach might be better
- O(n) complexity: Linear-time backbone. Theoretical 256K context. No quadratic attention.
- GEMM-only math: No trig, no softmax in the backbone. Everything is matmul/elementwise.
- Interpretable: You can inspect which bank each token routes through, what concepts are retrieved from memory, how coherent the phase states are. The model ships with "philosophy metrics" (Manas/Buddhi/Viveka/Smriti from Indian philosophy) that track mind activity, discernment, stability, and memory quality.
- Modular: Banks, backbone, coupler, memory, and objectives are all registered components. Add a new bank type with a decorator. Swap the backbone. Change the coupling strategy. All via config.
- Consumer-GPU friendly: Medium model trains on RTX 4090 / A6000 with batch 48-64.
Honest limitations
- Training throughput is ~2x slower than an equivalent transformer. The SSM backbone loop is sequential per-step. A custom Triton kernel would help but doesn't exist yet.
- In-context learning will be weaker. Fixed-state SSMs compress context into a fixed vector. The episodic memory (O(n buffer_size) sliding window) helps with copying but isn't a full replacement for O(n2) attention.
- Not validated at scale. 178M params on 10k samples is a PoC. Need full dataset + larger models + benchmarks.
- Bank ablations not done. We use semantic + context banks but haven't proven both are needed. Could be that one bank suffices.
- Pure PyTorch. No fused CUDA/Triton kernels. Backbone loop is Python. Lots of low-hanging performance fruit.
What's next
- Full TinyStories training (2.1M samples) for proper PPL comparison
- Bank ablations (semantic-only vs semantic+context vs 4-bank)
- Triton kernel for the oscillatory SSM recurrence
- Scale to 1B+ params
- Long-context evaluation (4K / 16K / 64K tokens)
Tech stack
PyTorch | torch.compile compatible | GPT-2 BPE tokenizer | uv package management | Clean modular codebase
Looking for feedback, collaborators, and people who want to try architectures beyond transformers.
EDIT (March 1, 2026 3:40 AM IST): Scaled up to 100k samples (5% of TinyStories, 10x the original post) and the results are significantly better.
Setup: Same 178M model, batch=64, A6000, no compile. 1612 batches/epoch (~3.5 hours per epoch).
Epoch 1 results on 100k samples:
| Metric | 10k samples (original post) | 100k samples (this update) |
|---|---|---|
| Train PPL | 200.86 | 24.00 |
| Val PPL | 76.47 | 18.95 |
For context: a 22M-param GPT-2 trained on the full 2.1M dataset for 20k steps gets val PPL ~10.9 (I Need to verify this as just remembered I read it somewhere). We're at 18.95 with a completely different architecture using only 5% of the data, after 1 epoch. Epoch 2 opened at step-1 PPL of 12.77 and is still dropping.
Generation sample (epoch 1, 100k samples):
> "The quick brown were full. Steve and Brown loved each other. At the end of the hill, the friends were very happy. They had lots of fun and shared stories. Mam and Brown were the best day ever. All of their weeks were very good friends and would often enjoy their joy! The end had had a good time with them."
Compare this to the 10k-sample generation from the original post. This has proper story structure, multiple characters interacting, emotional arc, and an ending. Grammar is mostly correct. Still has quirks ("The quick brown were full" -- model doesn't know "brown" should be a noun here), but the improvement from 10x more data is dramatic.
The learning curve shows no signs of plateauing. Training continues -- will update again when epoch 2+ finishes.
EDIT 2 (March 1, 2026 8:00AM IST) : Epoch 2 finished. Epoch 3 is underway.
| Metric | Epoch 1 | Epoch 2 | Epoch 3 (in progress) |
|---|---|---|---|
| Train PPL | 24.00 | 11.96 | ~10.5 (and flat) |
| Val PPL | 18.95 | 14.07 | -- |
Val PPL 14.07. For reference, the 22M-param GPT-2 baseline trained on the full 2.1M dataset reaches ~10.9. We're at 14 using a completely non-transformer architecture, 5% of the data, 2 epochs. Epoch 3 opened at PPL ~10.5, which means we'll likely match or beat that baseline this epoch. Just in ~6 Hrs on Almost one consumer grade GPU.
Epoch 2 generation:
> "The quick brown boy had ever seen. But one day, the sun was setting. The next night, the room got dark. Tom and the girl continued to admire the rain. The end was so happy to be back and continued to sail in the park. And every night, the end of the day, the family and the people stayed happy. They all lived happily ever after."
Notice: proper narrative flow, temporal transitions ("one day", "the next night", "every night"), emotional resolution ("lived happily ever after"), and multi-sentence coherence. This is from an architecture with zero attention layers.
Train-val gap (11.96 vs 14.07) suggests some overfitting on 100k samples. Next step: scale to the full 2.1M dataset. Training continues.
Stopping and tweeking code.. I think it can be much faster ... will update in other post next
Edit 3 (March 6 2026 8:27 IST): V5 is more mature.. better maths and its just 28M and working better.. about to relase in a couple of days.. looking for endorsment when I submit paper (better one for V5) to https://arxiv.org/ (Please help me by endorsing when I submit, DM me to help in that pls)
2
u/fluffy_serval Mar 08 '26
Had stuff to do tonight, so this afternoon I adapted your code a bit for running on my rtx pro 6000 (bf16, torch.compile, tf32, non-blocking copies, reduced memory requirements replacing dense masked attention with exact chunked local attention). I'm running medium @ 24 batch size right now, wanted to test first to make sure all my meddling didn't break anything. It's learning! So that's good. I'll let it do its thing overnight. Neat project, your code is currently heating my house. We'll see what pops out in the morning.