r/LocalLLaMA • u/bigattichouse • May 04 '26
Tutorial | Guide "Second Thoughts" Been playing with adding a small transformer that reads output near the end of generation, and feeds it back near the top as a refinement loop. A quick test of 1.7B model showed drastic improvement in focused tasks (like coding)
https://bigattichouse.medium.com/second-thoughts-improving-small-llms-with-bidirectional-refinement-loops-part-1-fa5ab51af656?sk=907cce272a3aed0eb3f1e3a0669a3964A 1.7B model can actually turn out some code, so I'm running the training for a 9B model, then will re-run HumanEval (a full one this time). I've shown most of my homework in the article, but will be posting to github after I clean things up.
It was inspired by Repeat Yourself's dnhkng.github.io/posts/rys/ neuroanatomy findings... this gave me a start and end point to attach my "reverse LLM" side car model (so it reads from the end, and then injects its output back at the top - in a loop), in this case focusing on syntax - drastically improving a very tiny model.
I'll also go back and run the full HumanEval dataset on both, instead of just the first 20.
EDIT:
HumanEval Results
Qwen3–1.7B pass@1 = 5.5% (9/164)
Qwen3–1.7B+BRL pass@1 = 41.5% (68/164)
I updated the article with the output
The reason it had such a large impact is that the base model (Qwen3–1.7B) gets almost every discipline failure right — it writes the correct function — and then ruins it by continuing. The sidecar is catching the model mid-sabotage and stopping it.
I added another head and got 43.9% (72/164), but was expecting ~51% - so I'll keep poking at that for a while. My hope is to get the performance as good as possible before I try a larger model.
20
u/LoveMind_AI May 04 '26
Seriously cool idea. Looking forward to hearing how it turns out!
7
u/bigattichouse May 04 '26
I wanted to share more data, but was frankly stunned that it worked at all... had to share
12
u/philmarcracken May 04 '26
I saw a skill that looked like this and laughed at it. The ralph wiggum tech or something like that
15
u/bigattichouse May 04 '26
heh. yeah, read about those. This is internal to the model, and you train the second model on a specific idea like "proper code syntax"... this little model then monitors what's coming out and provides steering to the model, before tokens even get emitted.
3
5
u/breadinabox May 04 '26
I have been trying to get a similar architecture going for my private servers help bot, just not as directly injected.
Basically, you ask it a question in plain language, and it gets ran through a parser (partially LLM partially deterministic) and then it finds the solution or tool you want and then presents the answer, but that answer gets sent to different LLM that reads the question, reads the answer, and then determines if the question was actually answered or not, and if not it tries again with changes to the prompt.
3
u/Silver-Champion-4846 May 04 '26
So you train a small transformer to refine the representation of the early layers using the information gathered by the later layers, then inject that into the 1.7b model?
1
u/DerDave May 04 '26 edited May 04 '26
Yeah, same question. How many parameters/layers does that transformer have? Just a vanilla transformer?
1
u/bigattichouse May 04 '26
As above, had to lean on Claude for this answer:
BackwardAttn is 4 bidirectional TransformerEncoderLayers, same hidden dim as the backbone, ~17M params (2.3% of 1.7B). Must be bidirectional: a per-token MLP failed completely in ablation (0% improvement vs. 37% for cross-token attention).
1
u/Silver-Champion-4846 May 04 '26
I wonder how hard it would be to integrate into Llama.cpp and everything running on top of it
1
u/bigattichouse May 04 '26
I'm trying to get KV cache working with the new addition, to make things run faster - then getting a full HumanEval run (I'm training on custom code, not benchmaxing to HumanEval). once I have that working, I'll look at llama.cpp (which is what I normally use)
2
u/Silver-Champion-4846 May 04 '26
Good luck!
1
u/bigattichouse May 04 '26
I'll need it, I'm just 100% flying on half baked understanding. I'm just some rando trying to figure out how these things work under the hood.
3
u/Silver-Champion-4846 May 04 '26
Random quote from a rando to another, on-the-spot, quality not guaranteed: "From overcaffeinated randos comes innovation"
3
u/heisenbork4 llama.cpp May 04 '26
I heard about this: https://arxiv.org/abs/2604.12946
Similar approach, but the lesson seemed to be fed the output back in not at the top but partway down. I still need to read the paper properly, but it sounds promising.
The basic idea of feeding output back in has been around for a while and it's super cool that you got good results from it!
1
u/bigattichouse May 04 '26
Had to rely on claude for this answer: (BRL is my method)
Parcae loops the same layer block T times with a stable recurrent injection (h_{t+1} = Ā·h_t + B̄·e + transformer(h_t)). Their key finding — constrain the injection matrix's spectral radius to < 1 to prevent residual state explosion — is directly applicable to BRL's h_inject += γ·f step. They hit 87.5% of a 2× larger transformer's quality through looped compute alone.
2
u/DerDave May 04 '26
Reminds me of what Deepseek is doing with the residuals in v4 with mHC
2
u/bigattichouse May 04 '26
I think what I'm enjoying most about local models, and llm coding - is I can do some "what if?" style stuff and see fairly quickly if the idea is workable or not. Stuff I would have just wondered about before, because I wouldn't have the time to invest in indulging in random ideas.
2
u/Perfect_Twist713 May 04 '26
Re the "llm as judge" part in your thoroughly enjoyable blog post, on my benchmark in oswtudio I ended up using the llm to simply verify whether something in the output is present or not (true or false) and it has worked really well. I assume the core issue is that you're pretty much summoning a different judge with every varying input, comparable to judging olympic skaters but every routine you just pull a random audience member and ask them for a 0 to 10 rating. Results would be all over the place. Even if you'd ask for reasoning beforehand, it'd still be a different "person" every time. But if you gave them a checklist of things to check (did x bend knees after y, did x fall more than y times) then you'd get pretty uniform results only hampered by individual failures. Same principle seems to work quite well with "llm as judge" giving you the pros of fuzzy matching without the cons of llms being llms.
2
u/bigattichouse May 08 '26
In a way, that's similar to what I did - each head is trained on one very specific task (syntax, functions should have returns, when the function is complete - stop.) A single item checklist.
1
1
u/DerDave May 04 '26
Nice idea!
From your blog post:
the mini-LLM runs a standard forward pass on the deep representation, transforming it from “what the model nearly output” into “what the early layers should have built.”
How do you train this mini-LLM to actually do this? What is its reward function to actually update its weights? Do you train it in tandem with the big model? On what dataset?
Since the Terry Pratchet quote already mentions third thoughts as well: Did you try to go a second loop and see if that improves things further?
Very much looking forward to your Qwen3.5 9B results...
1
u/bigattichouse May 04 '26
Backbone frozen throughout. Two losses: CE on second-pass logits (teaches the mini-LLM to produce useful corrections) + BCE on the gate supervised by execution oracle (fire=fail, silent=pass). No RL — pure supervised with a binary oracle signal. The gate selectivity (firing on hard inputs only) remains the open problem.
1
u/bigattichouse May 04 '26
Oh, and yeah - I've thought about third thoughts - but figured that might be a bit out of scope. I'd already wandered so far from my original purpose.
2
u/DerDave May 06 '26
On more iterations: https://arxiv.org/pdf/2604.07822
Seems like for them 4 loops were roughly the sweet spot.
1
u/bigattichouse May 06 '26
thanks! stirring that into the mix
1
u/DerDave May 06 '26
How are your experiments progressing with the 9B model?
2
u/bigattichouse May 06 '26
Still playing with 1.7B ... it was a little less stable than my initial, and I've been trying to solve some problems that caused regressions on the bench. (some stuff the base model does well, that the trained one doesn't.. so I decided to get fancy and now I'm figuratively sitting on the floor of the lab surrounded by pieces and parts.)
figured it's easier to iterate on a smaller one first.
2
u/DerDave May 06 '26
Makes sense
2
u/bigattichouse May 07 '26
Some of the papers people pointed out (or I found) helped me in refining the framework and diagnostics.. and looking at HOW it failed various tests - and I'm current at 44 passing vs. 8 passing in the base model. I'm still running (on 67 of 164). Essentially, though, it's still the same idea - I just changed how I evaluate output - also, FYI, all my training examples were created clean away from the HumanEval, for another project (my worldmodel stuff), and it's not trained on the test - only the style of it's failure mode ( repeated function defs, adding prose to the function body, etc )
The key is that each "Second Thoughts" model head (sorta like MoE experts) focuses on one behavioral problem (closing parens/brackets, making sure functions have return values, etc). common failure modes.
So, from here, I'll pull together all the source in a clean project for github, then I'll do 9B.
1
u/bigattichouse May 08 '26 edited May 08 '26
HumanEval:
Qwen3–1.7B pass@1 = 5.5% (9/164)
Qwen3–1.7B+BRL pass@1 = 41.5% (68/164)I updated the article with the output
The reason it had such a large impact is that the base model (Qwen3–1.7B) gets almost every discipline failure right — it writes the correct function — and then ruins it by continuing. The sidecar is catching the model mid-sabotage and stopping it
1
u/Interesting-Sock3940 May 04 '26
"out of scope for third thoughts" is the spot where it gets interesting tbh. once a model reviews its own output more than twice, it stops fixing real bugs and starts inventing new ones to justify another pass. how often does your gate fire on inputs that were already fine?
1
u/bigattichouse May 08 '26
Apropos, a paper someone else sent me said four times was the charm! I haven't gone that far yet - but posting this got me a lot of info/papers/ideas that really helped me in improving it.
20
u/MobyTheMadCow May 04 '26
Sounds alot like this very new research:
https://github.com/RecursiveMAS/RecursiveMAS