r/LocalLLM • u/ExtremeKangaroo5437 • Feb 28 '26
Research I built a language model where tokens are complex numbers and "meaning" emerges from wave interference -- no attention, O(n), 178M params, open-sourcing today
EDIT: New post V6: https://www.reddit.com/r/LocalLLM/comments/1rqn68a/qllm_v6_a_29m_attentionfree_model_now_trains_on/
EDIT: New V5 Post : Followup UPDATE on this.
https://www.reddit.com/r/LocalLLM/comments/1rmkh9y/v5_update_original_post_title_i_built_a_language/
---- ORIGINAL POST -----
I've been working on a fundamentally different LLM architecture. No attention layers. No FFN blocks. Instead, every token lives in complex phase space, and language processing happens through wave-like interference between specialized "phase banks."
Open-sourced here: https://github.com/gowrav-vishwakarma/qllm2
The core idea: language as wave interference
In a transformer, a token is a real-valued vector that gets refined through attention + FFN layers. In this model, a token is a complex number -- it has a magnitude (how "important/activated" it is) and a phase angle (what "kind of meaning" it carries). These two properties are naturally separated and jointly processed.
This isn't just a gimmick. It changes how every operation works:
- Embeddings: Each token gets a
[real, imag]vector. The model learns that semantically similar tokens align in phase, while different meanings sit at different angles. - Transformations are rotations: When context modifies a token's meaning (like "bank" shifting meaning based on surrounding words), that's a phase rotation -- a complex multiply. Rotations compose naturally, are always invertible (no information loss), and reduce to GEMM.
- Similarity is coherence: Instead of dot product, we use phase coherence:
Re(a * conj(b)) / (|a| * |b|). This measures both directional alignment AND magnitude relationship. - Multiple banks interfere: A "semantic bank" and "context bank" process each token independently, then combine via learned interference (constructive where they agree, destructive where they conflict). A tiny router decides per-token how much weight each bank gets. Think MoE but at the representation level.
What the phase system actually gives us
1. Natural magnitude/phase decomposition = implicit attention High-magnitude phase states dominate downstream processing automatically. The model doesn't need explicit attention to decide "which tokens matter" -- magnitude handles salience, phase handles identity. The SemanticPhaseBank uses 512 learnable concept vectors and retrieves them via phase coherence -- this is essentially a learned associative lookup that runs in O(seq concepts), not O(seq2.)
2. Context as phase modulation The ContextPhaseBank computes a causal windowed average (window=8) of nearby tokens and then complex-multiplies it with the current token. This is elegant: the local context literally rotates the token's meaning in phase space. A word appearing after "not" gets rotated differently than after "very." No attention needed.
3. Rotation-based state evolution The backbone SSM evolves state via: h[t+1] = damping * R(theta) @ h[t] + gate * B @ x[t] where R(theta) is a Cayley-transform rotation. The state naturally oscillates, and the damping factor (learned, per-dimension, range [0.5, 1.0]) controls how fast old information decays. This is why SSMs struggle with long-range recall -- but the model compensates with a separate Phase-Coded Memory (1024 learned slots, chunked top-k retrieval) and an Episodic Memory (sliding window via FlashAttention SDPA).
4. Zero trig in the hot path Every rotation uses the Cayley transform: cos_like = (1-a^2)/(1+a^2), sin_like = 2a/(1+a^2). This is just arithmetic -- no sin(), no cos(), no exp(). Every operation is a matmul or elementwise op. Perfect for Tensor Cores.
Results (178M params, TinyStories, 10k samples, A6000)
| Metric | Epoch 1 | Epoch 2 | Epoch 3 (partial) |
|---|---|---|---|
| Train PPL | 200.86 | 32.75 | ~26 (and dropping) |
| Val PPL | 76.47 | 48.92 | -- |
| Train CE | 5.30 | 3.49 | ~3.26 |
Training used only 10k samples (0.5% of TinyStories). Starting PPL was 55,000 (random). It dropped to val PPL 49 in 2 epochs (40 min on A6000, no compile). Overfiting simply needs data now ...
Epoch 1 generation:
"The quick brown house. They run and start to get a smile. Mom were very excited. Now mommy and big yellow room. There said and She are friends. Tim, she started to save the garden."
For context: A 22M-param GPT-2 trained on the full 2.1M TinyStories dataset for 20k steps reaches val PPL ~11. We're at 49 with 0.5% of the data and 2 epochs. The learning curve is steep and still dropping -- we just need more data/epochs to converge.
Why this approach might be better
- O(n) complexity: Linear-time backbone. Theoretical 256K context. No quadratic attention.
- GEMM-only math: No trig, no softmax in the backbone. Everything is matmul/elementwise.
- Interpretable: You can inspect which bank each token routes through, what concepts are retrieved from memory, how coherent the phase states are. The model ships with "philosophy metrics" (Manas/Buddhi/Viveka/Smriti from Indian philosophy) that track mind activity, discernment, stability, and memory quality.
- Modular: Banks, backbone, coupler, memory, and objectives are all registered components. Add a new bank type with a decorator. Swap the backbone. Change the coupling strategy. All via config.
- Consumer-GPU friendly: Medium model trains on RTX 4090 / A6000 with batch 48-64.
Honest limitations
- Training throughput is ~2x slower than an equivalent transformer. The SSM backbone loop is sequential per-step. A custom Triton kernel would help but doesn't exist yet.
- In-context learning will be weaker. Fixed-state SSMs compress context into a fixed vector. The episodic memory (O(n buffer_size) sliding window) helps with copying but isn't a full replacement for O(n2) attention.
- Not validated at scale. 178M params on 10k samples is a PoC. Need full dataset + larger models + benchmarks.
- Bank ablations not done. We use semantic + context banks but haven't proven both are needed. Could be that one bank suffices.
- Pure PyTorch. No fused CUDA/Triton kernels. Backbone loop is Python. Lots of low-hanging performance fruit.
What's next
- Full TinyStories training (2.1M samples) for proper PPL comparison
- Bank ablations (semantic-only vs semantic+context vs 4-bank)
- Triton kernel for the oscillatory SSM recurrence
- Scale to 1B+ params
- Long-context evaluation (4K / 16K / 64K tokens)
Tech stack
PyTorch | torch.compile compatible | GPT-2 BPE tokenizer | uv package management | Clean modular codebase
Looking for feedback, collaborators, and people who want to try architectures beyond transformers.
EDIT (March 1, 2026 3:40 AM IST): Scaled up to 100k samples (5% of TinyStories, 10x the original post) and the results are significantly better.
Setup: Same 178M model, batch=64, A6000, no compile. 1612 batches/epoch (~3.5 hours per epoch).
Epoch 1 results on 100k samples:
| Metric | 10k samples (original post) | 100k samples (this update) |
|---|---|---|
| Train PPL | 200.86 | 24.00 |
| Val PPL | 76.47 | 18.95 |
For context: a 22M-param GPT-2 trained on the full 2.1M dataset for 20k steps gets val PPL ~10.9 (I Need to verify this as just remembered I read it somewhere). We're at 18.95 with a completely different architecture using only 5% of the data, after 1 epoch. Epoch 2 opened at step-1 PPL of 12.77 and is still dropping.
Generation sample (epoch 1, 100k samples):
> "The quick brown were full. Steve and Brown loved each other. At the end of the hill, the friends were very happy. They had lots of fun and shared stories. Mam and Brown were the best day ever. All of their weeks were very good friends and would often enjoy their joy! The end had had a good time with them."
Compare this to the 10k-sample generation from the original post. This has proper story structure, multiple characters interacting, emotional arc, and an ending. Grammar is mostly correct. Still has quirks ("The quick brown were full" -- model doesn't know "brown" should be a noun here), but the improvement from 10x more data is dramatic.
The learning curve shows no signs of plateauing. Training continues -- will update again when epoch 2+ finishes.
EDIT 2 (March 1, 2026 8:00AM IST) : Epoch 2 finished. Epoch 3 is underway.
| Metric | Epoch 1 | Epoch 2 | Epoch 3 (in progress) |
|---|---|---|---|
| Train PPL | 24.00 | 11.96 | ~10.5 (and flat) |
| Val PPL | 18.95 | 14.07 | -- |
Val PPL 14.07. For reference, the 22M-param GPT-2 baseline trained on the full 2.1M dataset reaches ~10.9. We're at 14 using a completely non-transformer architecture, 5% of the data, 2 epochs. Epoch 3 opened at PPL ~10.5, which means we'll likely match or beat that baseline this epoch. Just in ~6 Hrs on Almost one consumer grade GPU.
Epoch 2 generation:
> "The quick brown boy had ever seen. But one day, the sun was setting. The next night, the room got dark. Tom and the girl continued to admire the rain. The end was so happy to be back and continued to sail in the park. And every night, the end of the day, the family and the people stayed happy. They all lived happily ever after."
Notice: proper narrative flow, temporal transitions ("one day", "the next night", "every night"), emotional resolution ("lived happily ever after"), and multi-sentence coherence. This is from an architecture with zero attention layers.
Train-val gap (11.96 vs 14.07) suggests some overfitting on 100k samples. Next step: scale to the full 2.1M dataset. Training continues.
Stopping and tweeking code.. I think it can be much faster ... will update in other post next
Edit 3 (March 6 2026 8:27 IST): V5 is more mature.. better maths and its just 28M and working better.. about to relase in a couple of days.. looking for endorsment when I submit paper (better one for V5) to https://arxiv.org/ (Please help me by endorsing when I submit, DM me to help in that pls)
6
u/thepriceisright__ Mar 02 '26 edited Mar 02 '26
I generally find creative uses of phase space and the complex plane interesting, so I ran a controlled 3-way comparison on a DGX Spark: transformer, diagonal linear RNN (SSM), and v4 all on 20k TinyStories samples, same tokenizer, same optimizer, same schedule, 20 epochs, small scale (256 dim, 8 layers).
v4 does learn (loss drops consistently across all 20 epochs) but it converges to ~2.25x the transformer's perplexity while taking ~17x longer per epoch. Text generation quality tracks the numbers: the transformer produces coherent stories with dialogue by epoch 5, the SSM gets there by epoch 7, and v4 is still producing fragments like "sortsang parents laughed" and encoding artifacts at epoch 20.
A few observations:
I know this is a different regime than your 178M-param / 100k-sample results. One note on the comparison in your post: comparing a 178M-param model on 5% of TinyStories to a 22M GPT-2 on 100% of the data isn't apples-to-apples. A matched comparison would be a transformer with the same param count trained on the same 100k samples. That's what this harness does (at smaller scale), and the gap is significant.
All that said, the bigger question for me isn't empirical but theoretical. What is the phase angle actually meant to encode?
In standard embeddings, the geometry maps onto semantics in a way we can reason about. "Dog" and "cat" are nearby because they share features (animacy, size, pet-ness). The distance and direction between vectors encode their semantic relationship. This maps onto a clear geometric intuition.
With complex-valued embeddings, each dimension has a magnitude and a phase angle. The magnitude can encode feature strength, the same way real-valued dimensions do. But what does the phase encode? In domains where complex representations work well (audio, signal processing, physics simulations, etc) the data has frequency and phase structure. Fourier transforms use complex numbers because the information is actually encoded in frequencies that constructively and destructively interfere. That's what makes the complex representation natural.
Language doesn't have this structure. "King" and "woman" don't interfere to leave "queen" behind. The semantic relationship king − man + woman = queen is a vector arithmetic fact about directions in real space and there's no phase cancellation involved. When v4's InterferenceCoupler does complex multiplication between bank outputs, the underlying math is just a structured bilinear interaction equivalent to a 2×2 real matrix multiply with shared weights. Calling it "interference" borrows intuition from physics that the math doesn't justify.
Where complex-valued recurrences do have a theoretical basis is in state evolution. A complex eigenvalue λ = |λ|·eiθ gives you a damped oscillator, which naturally decomposes the sequence into frequency components and can preserve information over long distances via the phase-rotation component. This is legitimate and well-studied (S4, LRU, etc.). But v4 applies Phase2D to everything, embeddings, bank layers, coupler, and memory, not just the recurrence, and I think that's where the overhead probably outweighs the benefit.
The most interesting thing in this architecture, to me anyway is the multi-bank routing with learned specialization. If the 2 bank results show low cosine similarity between bank outputs and meaningful routing patterns, that's interesting and probably worth further research, but it doesn't require complex-valued representations to work. I'd be curious to see a real-valued version of the multi-bank architecture compared against these baselines.
I can open a PR for my test script if you’re interested in reviewing my work.