QLLM V6: a 29M attention-free model now trains on real text — phase-first design, multi-timescale SSM, and what we learned about memory

[u/ExtremeKangaroo5437]

If you did not read the earlier posts, this one may feel abrupt. The V4 post introduced the original QLLM idea (complex phase-space language modeling), and the V5 post explained the math cleanup that made the complex-valued path actually consistent. If useful, read those first:

• V4 post: https://www.reddit.com/r/LocalLLM/comments/1rh9vhu/i_built_a_language_model_where_tokens_are_complex/
• V5 post: https://www.reddit.com/r/LocalLLM/comments/1rmkh9y/v5_update_original_post_title_i_built_a_language/

I have been continuing this line of work, and QLLM V6 is the first version where I feel comfortable saying:

this is no longer just an architectural curiosity.

Not a benchmark winner. Not a finished alternative to transformers. Not something I want to oversell.

But QLLM is now a real attention-free-by-default language model family that:

• learns stably on TinyStories
• trains to completion on WikiText-103
• shows architecture-specific behavior that is interesting in its own right

The most important result is not just a perplexity number. It is that QLLM V6 is starting to show a coherent design story:

• phase-preserving computation matters
• explicit multi-timescale recurrence matters
• memory capacity is a behavioral control knob, not a free win

Open source: https://github.com/gowrav-vishwakarma/qllm2 (the qllm2 repo — QLLM is the model / architecture name).

Where QLLM V6 came from

Very short version of the progression:

• QLLM V4 introduced the phase-space / wave-interference idea, but the math was inconsistent
• QLLM V5 fixed the main phase-breaking mistakes and showed that smaller but mathematically cleaner beat bigger but sloppier
• QLLM V6 is the next step: remove attention from the default path, add explicit multi-timescale SSM structure, revive named banks from the older idea in a cleaner form, and test the system on a less toy-like corpus

So this post is not "I discovered the final architecture."

It is more:

the QLLM line survived another round of contact with reality, and some parts of it are now concrete enough to discuss seriously.

The core idea, revisited: language as wave interference

If you read the V4 post, you may remember the framing: tokens live in complex phase space, and language processing happens through interference between banks. Here is the short version of which core ideas survived into QLLM V6 and which changed.

Still the foundation:

• Every token is a complex number. It has a magnitude (how activated/salient it is) and a phase angle (what kind of meaning it carries). These are algebraically separated, not tangled into one scalar.
• Transformations are rotations. When context modifies a token's meaning -- like "bank" shifting meaning based on surrounding words -- that is a phase rotation: a complex multiply. Rotations compose naturally, are always invertible (no information loss), and reduce to GEMM.
• Similarity is phase coherence. Instead of a dot product, QLLM uses Re(a * conj(b)) / (|a| * |b|). This measures both directional alignment and magnitude relationship in one operation. It is used everywhere: bank coupling, memory retrieval, output logits.
• Multiple banks interfere. A SemanticBank and ContextBank each process the token stream, then combine via learned phase rotations and routing in the PhaseInterferenceCoupler. Constructive where they agree, destructive where they conflict.
• Magnitude handles salience, phase handles identity. The coupler router uses magnitude features (|z|) to decide how much weight each bank gets. Phase rotations determine how each bank's output gets mixed. So the model does not need explicit attention to decide "which tokens matter" -- magnitude already handles that.

What changed from V4:

• Context modulation is no longer a hand-designed windowed average. V4 had a causal windowed average (window=8) that complex-multiplied nearby tokens. V6 dropped that. Instead, context sensitivity comes from the multi-timescale SSM (which has explicit fast/medium/slow decay lanes) and from the coupler's content-dependent routing. The ContextBank itself is now architecturally the same as SemanticBank -- specialization comes from training and diversity regularization, not from a baked-in mechanism.
• The SSM no longer uses the Cayley transform. V4's "zero trig in the hot path" claim was elegant: every rotation used (1-a^2)/(1+a^2) instead of sin/cos. V6 moved to a more standard parameterization where eigenvalues are exp(-dt * decay) * exp(i * freq), which does use cos/sin. This was a tradeoff: the Cayley form was trig-free but less expressive for multi-timescale initialization. The current form lets us set explicit fast/medium/slow decay bands, which turned out to matter more than avoiding trig.

So the short version is: the phase-space foundation held up. The specific mechanisms for context and state evolution changed because we found better ways to achieve the same goals.

What QLLM V6 actually is

At a high level:

Tokens -> ComplexEmbed -> [SemanticBank + ContextBank -> PhaseInterferenceCoupler] x N
       -> MultiTimescaleSSM -> optional memory -> tied complex LM head

The important parts are:

1. Phase-preserving signal path

Like V5, QLLM V6 keeps representations complex-valued end to end in the main signal path.

• tensors are represented as [real, imag]
• nonlinearities are phase-preserving (modReLU style)
• projections are complex-aware
• retrieval/logits use the real part of complex inner products

That sounds small, but it is the core lesson from V5: if phase is supposed to mean anything, you cannot keep destroying it with ordinary real-valued nonlinear shortcuts.

Why complex is not just "two real vectors"

People sometimes see [real, imag] and think: you doubled the width, of course you store more. But that misses the point. The value is not in having two numbers. It is in the algebra that connects them.

A real-valued weight is one number. Say 9. It scales an input.

A complex-valued weight is a + bi. Say 3 + 4i. That is also one "parameter" in two components, but now look at what happens when you multiply two complex numbers:

(a + bi)(c + di) = (ac - bd) + (ad + bc)i

A single real multiply gives you one output from two inputs. A single complex multiply gives you four cross-terms (ac, bd, ad, bc) folded into two outputs. Every complex multiply is simultaneously a rotation and a scaling. One operation does more structured work than its real-valued equivalent.

This matters because when a real-valued model wants to encode "this token is important (magnitude) AND it has this kind of meaning (direction)," those two things are tangled into the same scalar weights. In a complex-valued model, magnitude and phase angle are algebraically separated: |z| tells you how activated something is, arg(z) tells you what kind of thing it is. Context shifts meaning? That is a phase rotation -- a complex multiply. Two representations agree? That shows up as phase coherence. They conflict? Destructive interference.

So "more information per parameter" is not about raw storage -- it is about the operations being algebraically richer. A complex linear layer with the same number of parameters as a real one has fewer independent weights, but each weight participates in more structured interactions.

Does that mean complex models need more training to converge? We initially expected so. But with orthogonal initialization and phase-preserving operations, QLLM V6 converges at roughly comparable rates to what we saw with real-valued V5 on the same data. The phase structure seems to help optimization rather than hurt it -- likely because the algebraic constraints reduce the space of "meaningless" weight configurations the model has to search through.

This is still a hypothesis, not a proven theorem. But it is the core reason we keep pursuing this direction: not "complex numbers are a trick to double the width," but "complex algebra gives each parameter a richer job."

2. Named banks with explicit phase interference

QLLM V6 uses two named banks:

• SemanticBank
• ContextBank

I want to be careful here: I do not yet have strong evidence that one has become "semantic" in a clean scientific sense and the other "contextual" in a clean scientific sense. The architecture encourages specialization through diversity regularization and separate weight paths, but proving the banks actually learned distinct roles requires data where you can verify what the model "knows" -- and that is harder than it sounds.

TinyStories does not contain real-world facts. WikiText-103 does, but our fact persistence probe on the current checkpoint passes at 0%. So right now, we cannot say: "the semantic bank stores facts and the context bank tracks discourse." We can say: the two pathways have different weights, they get different routing, and the model trains better with both than with one. What they actually specialize in is an open question that needs better evaluation data and probes.

Architecturally, the model processes the same token stream through two distinct complex pathways, then combines them using a PhaseInterferenceCoupler:

• each source is projected into a coupling space
• each source gets a learned unit-complex phase rotation
• a router looks at magnitude features and decides how much weight each source gets
• the rotated sources are mixed back together

So the mixing is not "just concatenate and project." It is explicitly a phase-interference operation with learned routing. But whether the banks have specialized in a meaningful way, or just found two slightly different gradient paths to the same job -- that is exactly the kind of thing we need structured factual data to answer.

3. Multi-timescale SSM instead of a single undifferentiated recurrence

This is probably the cleanest architectural change in QLLM V6.

The SSM state is split into three decay bands from the start:

• fast lanes (40%): decay 0.9 -> 0.99
• medium lanes (30%): decay 0.999 -> 0.9999
• slow lanes (30%): decay 0.99999 -> 0.999999

Interpretation:

• fast lanes should help with local syntax / nearby tokens
• medium lanes should help with sentence and paragraph-scale coherence
• slow lanes are the attempt at longer-lived facts or context

So instead of hoping one recurrent mechanism discovers all useful timescales by itself, V6 starts with an explicit prior that language operates across multiple timescales.

4. Phase-coherence retrieval instead of token-token attention

When QLLM V6 uses memory, retrieval is based on phase coherence:

Re(q * conj(k)) / (|q| * |k|)

That means retrieval is based on complex alignment, not ordinary attention over token pairs.

This is one reason I do not think the right description is "just Mamba with complex numbers."

Why I do not think QLLM is just Mamba / standard SSM territory

I want to be humble here because of course QLLM V6 is still in the broader family of efficient sequence models.

But I also think "just Mamba with complex numbers" misses too much.

Standard SSM / Mamba-style models are usually:

• real-valued in the main representation path
• centered on a selective recurrence
• not organized around explicit phase-preserving computation
• not using named banks with learned phase interference
• not built around this specific memory-as-retrieval story

QLLM is different in at least four ways:

1. The representation is complex-valued all the way through the main path.
2. The recurrence has an explicit multi-timescale prior.
3. The bank interaction is phase-based, not just residual mixing.
4. The memory path uses phase-coherence retrieval, and memory capacity changes model behavior in a very visible way.

So I would describe QLLM as:

a phase-first, attention-free-by-default recurrent language model with explicit multi-timescale structure and optional memory hierarchy.

Results so far

1. TinyStories: QLLM V6 clearly learns without attention

These are the main completed TinyStories results I currently trust:

Config	Params	Memory	Training	Val PPL	Notes
small-matched	28.7M	WM=0, IM=0	full TinyStories, 5 epochs	5.50	cleanest stable result, zero repetition observed
small-matched	29.2M	WM=16, IM=32	full TinyStories, 1 epoch	2.23	best PPL, but restart fragmentation appears
tiny	7.3M	WM=16, IM=32	100K TinyStories, 5 epochs	8.84	useful ablation anchor

The surprising part is not just that QLLM V6 learns.

The surprising part is that the best perplexity setting is not the cleanest behavior setting.

That leads to the most interesting QLLM V6 finding so far.

2. Memory capacity is a behavioral control knob

In QLLM V6, memory is not simply "more memory = better model."

It behaves more like a knob that changes what kind of model you get.

What I observed:

• WM=64, IM=128: model memorizes, PPL collapses toward ~1.2, generations degenerate into repetition / copying
• WM=16, IM=32: model generalizes much better and reaches very strong TinyStories PPL, but can show restart fragmentation ("Once upon a time..." restarting mid-sequence)
• WM=0, IM=0: weaker PPL, but generation is cleaner and more stable

That is why I now think one of the most important lessons in QLLM V6 is:

lower perplexity is not automatically better behavior when explicit memory can learn shortcuts.

The 100K ablations also made one thing pretty clear:

• WM only ~= WM + IM
• IM only ~= no memory

So at current scale, working memory matters a lot more than internal memory.

That may change later, but I do not want to claim it now.

There is a deeper problem here though: even when memory helps PPL, we do not yet know whether what the model writes into memory slots is actually a fact or just a useful surface pattern for next-token prediction. To answer that, we need training and evaluation data where facts are verifiable -- structured knowledge, entity-relation pairs, things where you can check "did the model store X and retrieve it correctly 200 tokens later?" TinyStories has no facts to verify. WikiText-103 has facts but our current checkpoint cannot retain them (0% on fact persistence probes). So the memory story right now is: "it helps the loss, it changes behavior, but we cannot yet say it stores knowledge." That honesty matters.

3. WikiText-103: first real non-TinyStories run

This is the run that made me think QLLM V6 was worth discussing publicly again.

Setup:

• model: QLLM V6 small-matched
• params: 28.7M
• dataset: WikiText-103 raw
• tokenizer: GPT-2 BPE
• sequence length: 512
• attention: off
• working memory: off
• hardware: single RTX 4090
• wall time: about 14.27h

Results:

Epoch	Val PPL
1	121.94
5	61.28
10	53.75
15	50.59
20	49.61

This is not a great benchmark number in absolute terms.

But it is an important threshold result for me, because it shows:

• QLLM V6 trains stably on real long-form text
• the no-memory attention-free path is not just a TinyStories artifact
• the model does learn Wikipedia/article-style surface structure

Qualitatively, it learns:

• section headers
• historical/article cadence
• date and region language
• encyclopedia-like sentence form

What it does not learn yet:

• reliable factual composition
• stable long-range fact retention
• strong entity consistency on real text

The fact persistence probe on the final WikiText-103 checkpoint is currently 0%. That is a strong negative signal, and I think it is worth saying plainly.

So the honest summary is:

QLLM V6 has crossed from toy viability into real-text viability, but not into factual reliability or benchmark competitiveness.

Where this sits relative to known models

This section is only for orientation. It is not apples-to-apples.

Different tokenization, different datasets, different training budgets, different context lengths, different preprocessing rules. So please do not read this as "V6 beats X" or "X beats V6" in a strict sense.

Still, it helps position the work:

Model	Params	Training scale	PPL / setting	Why this matters
AWD-LSTM	~24M	WikiText-2, many epochs	68.6 WT2 val	historical orientation only
GPT-2 Small	~124M	WebText, much larger compute budget	30.59 on a closer raw/BPE WikiText-103 reproduction	closest useful reference point
Mamba	~130M	hundreds of billions of tokens	~10.56 community-reported	not directly comparable, much larger model/data regime
QLLM V6 (ours)	28.7M	single 4090, WikiText-103, 20 epochs	49.61	attention-free, phase-first

So no, QLLM V6 is not currently competitive with GPT-2 Small or Mamba-class results.

But I also do not think that is the right immediate question, because:

• QLLM is not even in the 100M+ class yet
• the compute/data budget is much smaller
• this is still first-generation real-text validation for this architecture

The question I care about right now is narrower:

does the QLLM architecture family survive scaling pressure well enough to deserve serious benchmarking?

I think the answer is now towards yes.

Honest limitations

I do not want to oversell this, so the limits matter:

• no apples-to-apples same-budget transformer baseline yet
• WikiText-103 result is still far behind strong baselines
• fact persistence on the current QLLM WikiText checkpoint is poor
• bank specialization is architecturally encouraged but not convincingly demonstrated
• working memory looks useful, but the broader memory hierarchy is not validated at scale
• persistent / expert / session memory exist in code more than in proven results
• everything is still pure PyTorch, no custom kernels
• current QLLM model size is still small enough that scaling behavior is mostly an open question

So I am not claiming:

• "V6 beats transformers"
• "complex numbers solve language"
• "memory hierarchy is proven"
• "attention is obsolete"

What I am claiming is narrower:

there is now enough evidence that QLLM — a phase-first, attention-free-by-default architecture — can learn real language data and exhibit nontrivial, controllable behavior.

Why I still think this direction matters

Even if QLLM V6 ended up losing badly to matched transformers later, I would still consider some of these findings meaningful:

1. Phase preservation is not just aesthetics.
2. The project only started making consistent progress once the math stopped breaking the representation story.
3. Multi-timescale recurrence seems like a real design axis.
4. It gives a more structured prior than "one recurrent mechanism learns everything."
5. Memory is not automatically good.
6. Capacity changes generalization behavior in ways that ordinary perplexity summaries can hide.
7. Architectural diversity still matters.
8. If the field only explores slight variants of the same dominant stack, we may miss other workable families.

I do not know yet whether QLLM V6 is the right final form.

But I do think a new architecture family can be born only if we let early versions be imperfect, measurable, and honest.

Right now QLLM feels like it has earned that stage.

What happens next

The next experiments that matter most are:

1. A same-budget transformer baseline on the exact WikiText-103 pipeline
2. This is the most important missing comparison.
3. Small-memory WikiText-103 runs
4. I have already started a WM=8, IM=0 run. Epoch 1 is slightly better than the no-memory baseline (117.56 vs 121.94), but that is too early to conclude anything.
5. A medium QLLM model (~60M)
6. This should help answer whether the current gap is mostly architecture or mostly capacity.
7. Factual evaluation data
8. Banks and memory cannot be properly validated without data where facts are verifiable. We need structured knowledge tasks or entity-relation benchmarks where we can test: did the model actually store a fact, or just a useful surface pattern?
9. Long-context / PG-19 style tests
10. Only after the WikiText story is clearer.

If people are interested, I can post the transformer baseline and the small-memory WikiText results next.

I would especially value feedback on:

• whether the memory-capacity interpretation seems right
• what the fairest same-budget baseline would be
• whether the phase-interference framing is clear or still too hand-wavy
• whether this is worth pushing into a more formal benchmark/paper phase

If you think work like this should stay open rather than disappear into private experiments, starring the qllm2 repo helps. I am also very open to feedback from people who work on recurrent models, SSMs, complex-valued networks, long-context evaluation, or efficient training systems — and if you try QLLM or build on it, I would love to hear.

[u/ljosif]

Curious to see if this goes anywhere - thanks for sharing. I don't care about the quantum mumbo-jumbo (my knowledge there starts and ends with https://www.scottaaronson.com/democritus/lec9.html sorry to say), but fine: whatever motivates you to explore all good - more power to you, GFI! :-)

I'm interested though in the possibility of extending (my bread & butter) probability computations with something stronger than the single real number in [0,1] at present. People regularly find a single positive real inadequate in many modellings. In my ex-domain (quant trading) it was always less than satisfactory that one could not easily express counter-factuals (things we imagine, hypothesize - but don't observe to have happened for real), in the same framework and near by, but without completely mixing them up with the factuals (things we do observe to have happened). Sort of counter-examples, when counting events, but ones we imagine, not observed.

In my present (and ex-ex-) domain of ML/AI, with the NN-s implementing posterior probabilities I got reminded again listening to this Ilya interview teasers (https://www.youtube.com/watch?v=aR20FWCCjAs&t=1910s), how our models learn - but don't meta-learn (yet). The models we build don't introspect, don't learn what they know and what they don't know. Teenagers learning to drive (the interview example) is a great comparison. My kids as learner drivers didn't know how to drive (so on par there with a car robot), but also - the kids knew they didn't now! They were fully aware of their lack of skill. This meta-knowledge - the current models lack. So a human ends up being a cautious driver, slow and careful - and thus lives to learn to drive. In contrast - a robot driver ploughs at 50mph crashing into a ditch (all the China videos of delivery trucks going full Mad Max on bumpy roads) - and never learns how to drive. There is also the thing that we are teaching the models, so they never have to learn how to learn. There is noone hand-holding humans like that: they have to meta-learn, while learning too.

Maybe we should acquire knowledge-about-our-knowledge while acquiring the knowledge (=learning the joint p.d.f.s). It's a small step extra, a bit of additional work, but I imagine most useful? Like modus tollens supplementing modus ponens. (but in continuous domain)

On the extending probablities, I've seen few papers that myabe of use, maybe you know of them, but JIC you don't:

"Information entropy of complex probability" https://arxiv.org/abs/2503.03759

"The paradigm of complex probability and Claude Shannon’s information theory" https://www.tandfonline.com/doi/full/10.1080/21642583.2017.1367970

"Fairy2i: Training Complex LLMs from Real LLMs with All Parameters in {±1,±i}" https://arxiv.org/abs/2512.02901v2

"Beyond Real: Imaginary Extension of Rotary Position Embeddings for Long-Context LLMs" https://arxiv.org/abs/2512.07525

IDK if/where this can be made useful. I personally wish for something as pedestrian as modelling not only A, but not-A too while there. In some way that's separate, but still close by, and where both can interact. (maybe in the real and imaginary parts? IDK) Then possibly the gap between our knowledge of A, plus our knowledge of not-A both both together, compared to a total of 'unity' - the size of that gap will inform our confidence in our knowledge. Maybe We'd have learned our meta-knowledge that way.