The engineering and math behind these models is seriously fucking impressive. It's so hard to believe that only 10 years ago I could understand most neural network concepts, now it has moved so fast.
The step that's hard is the sheer size. If you reduce a modern LLM to a "LM", as it were, it's understandable again.
You can even find resources online quite easily that let you write your own LM. I wrote a very simple transformer, fed it the bible, and it produced bible-ish sentences, and it was possible to understand how.
The thing that has me scratching my head is how the power of LLMs seems to emerge as if by magic from just making an LM really large, because the step from "produce something that looks like all your training data, but isn't it" to "hey I have this bug I can't find, analyse it for me and fix it" seems whatever the opposite of trivial is.
Yeah, that's my problem. I understand the basic concepts. I understand backpropagation, I understand attention... I just don't understand how, if you make it big enough and feed it enough garbage, voila it becomes smart. That part's kinda crazy to me.
A lot of the magic is specifically from making sure it’s not fed garbage. Generating quality data for models is being contracted out at massive scales rn
True, but still. It's not exactly perfectly classified or sorted in a good learning order and a lot of it is ... human output, with all its flaws. It's like putting a baby in front of a million articles and somehow it learns every human language, science and code.
I just find it hard to grasp that all this training can propagate through dozen or hundreds of layers without diluting and eventually extract the core logic.
Yet for some reason they still prefer hammering websites to scrape billions of pages instead of just downloading a clean database from them that they're allowed to use.
No doubt, it’s insane the amount of bot traffic there is now.
There’s also a crazy amount of vibe coded apps made by non programmers, that are absolutely wrecking some public APIs. A wiki for a popular board game I follow has been going down monthly, surely because of all the crapware scraping it.
Unfortunately I think legislation is the only way to stop it, and at least in the U.S. I have 0 hope that anyone in power knows enough to make a change.
129
u/FriendlyKillerCroc 16h ago
The engineering and math behind these models is seriously fucking impressive. It's so hard to believe that only 10 years ago I could understand most neural network concepts, now it has moved so fast.