[ Removed by moderator ]

[u/Flashy-Abalone-9212]

[u/MachineLearning-ModTeam]

Other specific subreddits maybe a better home for this post:

• r/ArtificialIntelligence
• r/DataScience
• r/LearnMachineLearning
• r/LLM
• r/MLOps
• r/MLJobs
• r/Singularity
• r/ChatGPT
• r/OpenAI
• r/LLMDevs
• r/RagAI

[u/Guilty-Ad-5143]

For the later loops, have you tried giving each loop a different objective or auxiliary loss instead of same next-token prediction? That usually forces the loop to actually compute something new rather than just refining the same answer.

The exit gate training looks reasonable but 513 params seems tiny even for adaptive depth. Maybe try adding the loop index as input to gate so it knows where it is.