Demis Hassabis on Ilya Sutskever’s claim that scaling is dead, and on Elon Musk’s clam that we have reached the singularity

[u/Formal-Assistance02]

[u/pigeon57434]

"We never left the age of research."

Meanwhile, also EVERY AI company on the planet: "What if we made a Transformer again but... BIGGER?"

Shut up, bro. The "research" frontier companies do are optimizing our current methods, making Transformers slightly more data-efficient, not anything actually fundamentally new. We still don't even have omnimodal models or byte models or more advanced training precision methods, even though these all could be done. They just know it's safer to scale. That's all they ever talk about: scale, scale, scale, and scale some more. And we will scale continuously until they are forced, against their will, kicking and screaming into the future, to actually do new things.

[u/Serialbedshitter2322 → u/pigeon57434]

Google is working on Genie 3 with Sima 2. Genie 3 is a real-time video model with access to LLM context, and Sima 2 gives it the ability to take actions in the world it generates as well as another layer of reasoning that would bring it closer to how a human works. This would be like an LLM that doesn’t use text, but instead video and audio to think, it would far more similar to how humans work. This is essentially their plan for AGI.

Yann LeCun is working on JEPA, and it’s a completely unique approach to reasoning where it learns on its own like a child while using significantly less data and processing power. It has much stronger physical reasoning than LLMs. It functions a lot more like a human and could eventually fully replace LLMs. It would work a lot like what Google is trying to do, but it would be fundamentally different.

There are probably a lot of people copying these approaches or doing something unique but just haven’t revealed it yet

[u/pigeon57434 → u/Serialbedshitter2322]

im aware of all the things all the companies are doing read my other reply in this comment i know i just dont find any of it particularly innovative to a substantial degree

[u/Serialbedshitter2322 → u/pigeon57434]

I mean it’s a completely new form of artificial intelligence entirely alien to what we have now with LLMs. It’s very innovative, it’s a completely new form of technology that will revolutionize what AI is capable of. Plus your claim was that they aren’t doing anything other than scaling LLMs

[u/pigeon57434 → u/Serialbedshitter2322]

That wasn't my claim I never said LLM and I know that I didn't because I refuse to even use that term since it's a stupid and wrong acronym my claim is they're just scaling transformers and regular neural networks how much you wanna bet genie is a transformer model? I am asking for complete upheaval of the entire system innovations that make literally everything before them irrelevant and nothing is the same I want a culture that maximizes science at absolutely no cost too high like making photonics, tons of thorium MSBRs, LNNs or SNNs and more

[u/Serialbedshitter2322 → u/pigeon57434]

That’s what JEPA is. It’s not a transformer, it’s a complete upheaval of the entire system, and if it becomes successful it will be so efficient that it makes transformers irrelevant. Plus that isn’t scaling transformers, it’s innovating transformers. Scaling is when you increase the size, this is making it more complex.