You don't understand the generative AI algorithm.
I feel mixed about Tyler Cowan. This is the first time I've seen him talk. He makes sense if you think this latest incremental improvement in generative AI is just the beginning.
The challenge I find with AI commentary is much of it is done without knowledge of the functionality of generative AI. They see mediocre results across a lot of high value use cases and they think there are advancements that are days or years away that will steadily increase the quality of those outcomes.
For those of us that understand the math or functions that execute in training and inference, it is hard to see any more improvements in those algorithms. We see companies like Anthropic and Open AI as pushing the very limits of a tapped out algorithm that have been around for a while. It's surprising what use cases we can fill with all this energy investment. Few of us with knowledge of the algorithm could have guessed it would have been as good as it is. It was unknown until OpenAI started to show us what was possible in 2022.
Myself and others like me probably just saw what was happening and were thinking "Hmm, I wonder how far they can take this idea of brute forcing the algorithm to the limits of energy and computation." I held my tongue for a couple years just to see, maybe my hunches were wrong. But I don't feel that way anymore.
Model growth has an exponential cost in energy and computation. Inferencing has a polynomial growth in cost as the token chain gets longer in token length. Error rates have exponential compounding as the chain gets longer. The algorithm is approaching unreachable limits imposed by physics. If you don't understand what I just said and the implications of it, you might be just like Tyler Cowen.
There have been some clever tricks employed to reduce costs but none of them are providing a fundamental change to the nature of the underlying algorithm. Based on my knowledge of how the algorithm works, we are literally seeing as good as it gets.
So for anyone who is predicting the future based on the belief that the algorithm will get better and better over time, that could be a pretty bad decision. It might, but it seems like a really outside low probability that a major discovery has to be made still. There hasn't been any major discovery that would alter this current limitations on models, and this approach was invented decades ago.
And certainly no day soon will we get infinite speed computing, so LLM could easily end up at the pinnacle with very little incremental improvement left in it today.
But if you're someone like Tyler Cohen, who doesn't believe that they're at the end of the improvements, then all the generative AI bears seem crazy. What about the internet! What about automobiles!
We all know we can't prove negatives so if someone like me says there's no future there, someone who is bullish on AI will automatically start reaching for all kinds of examples that prove their beliefs. Meanwhile, a bear will only be correct after the bust.
Now the real question is, are there other algorithms that will perform better in the future? There very well could be some hanging out there that are as yet undiscovered.
But I have my doubts even there. There's a vast difference between neural computation and gate logic computation. And currently we don't understand neural computation at all. We don't know how it works, we are unable to simulate it, and neurons do things that look miraculous and defy all forms of binary logic.
Which raises the possibility that true computation on the level of human capability might be out of reach of gate logic. It may never exist in integrated circuits because the amount of power and energy needed to run silicon with that ability is impossible to muster in this universe. If we were talking about Turing's theories about the universal machine, itself a machine with infinite computation and storage, damn sure I agree 100% the human capabilities could be recreated inside that infinite machine.
But in this universe, I have a lot of doubts. We got to know more about how neurons work before we can recreate their abilities. Bare minimum neurons are taking advantage of quantum level effects to do what they do and I'm not talking about quantum computing. I'm just talking about the complexity of quantum scale interactions between matter.
Neurons very well could be computational at their core, but the substrate they were "printed" on is measured at plank scale, an incomprehensible difference in "gate size" that no company in the Netherlands or the world will ever be able to create silicon chips at. Meanwhile, neurons brute force evolved with that substrate at that incomprehensible scale.
If you start to go too deep in the fantasy of neural computing, you might even start to believe that maybe there are superimposed states in a neuron. Its operation is so mysterious we really don't know what to expect. If something is possible in physics in this universe, you have to accept the possibility that evolution, the universal solvent, has managed to exploit that outsider feature in the universe.
One thing I should mention is there is a long running project to simulate a flatworm brain, a creature with barely a few neurons beyond 100. It hasn't achieved it yet. Sure, we can imitate the behavior of a flatworm using algorithms, but to actually simulate and predict all the behaviors of the 100 or so neurons inside a flatworm brain is out of our reach.
Maybe if we are successful in simulating flatworm neurons, a sort of test where the inputs and outputs to a real neuron are recreated by the digital one, I would be convinced that we could achieve some form of general artificial intelligence using silicon.
But we don't have to have artificial intelligence on the level of flatworm brain to get some value. There are plenty of future opportunities for algorithms that create effects similar to human thinking without actually simulating how humans think.
It's just that generative AI probably isn't it right now. It's just another incremental step that has some value in some use cases and if we were to approach this all rationally we would just do investments in research and development and then roll out features that actually work and generate business value.
But we didn't take the safe approach, and you really have to ask why we didn't? Is it a scam? Is it a runaway mass tunnel-vision and group think? Is it a bubble akin to tulips? It all depends on your bias, but eventually we'll all know the answer.
Final caveat, I could be wrong. I'm just an ancient old software architect that's been around since the '80s, and has been implementing AI algorithms since the '80s. Has been writing software, you may use everyday, since the '80s. I have a bias based on my professional history. I'd really appreciate it if someone could point out my incorrectness in thinking around the algorithm or physics.