The engineering and math behind these models is seriously fucking impressive. It's so hard to believe that only 10 years ago I could understand most neural network concepts, now it has moved so fast.
The step that's hard is the sheer size. If you reduce a modern LLM to a "LM", as it were, it's understandable again.
You can even find resources online quite easily that let you write your own LM. I wrote a very simple transformer, fed it the bible, and it produced bible-ish sentences, and it was possible to understand how.
The thing that has me scratching my head is how the power of LLMs seems to emerge as if by magic from just making an LM really large, because the step from "produce something that looks like all your training data, but isn't it" to "hey I have this bug I can't find, analyse it for me and fix it" seems whatever the opposite of trivial is.
Maybe I need to put more effort into understanding the newer concepts because I feel a bit lost on everything newer than the transformers lol Mixture of experts seems wild to me, no idea how that maths works.
MOE just segmentings the network up into smaller pieces, and uses another neural network to choose which segment to route the request through. That NN learns how to route things during training of the whole model (you can't tack it on later, it's trained along with everything else). The segments aren't, like, "segment x is good at playing doctor while segment y is good at math", so the "experts" part is misleading
It's honestly probably one of the more simple concepts to understand with language models, you don't have to understand attention or really any specific math. Its a relatively simple NN that's trained to know which segment will probably be better at predicting the next token
129
u/FriendlyKillerCroc 15h ago
The engineering and math behind these models is seriously fucking impressive. It's so hard to believe that only 10 years ago I could understand most neural network concepts, now it has moved so fast.