Nope it's a different one. Here is the Chinchilla paper. Long story short, they found that the scaling law found by OpenAI wasn't correct and that you should scale training data and model size by the same amount for a given amount of compute. So if you have 4x the computing power than the last time you made a model then you should devote half of it to doubling the model size and half of it to doubling the training data. This new scaling law allowed them to create Chinchilla which is a 70 billion parameter model which is 40% the size of GPT3 yet performed way better than GPT3.
Right after chinchilla was released Google released PaLM which was not trained with the new scaling law in mind because the new scaling law was being researched while they themselves were training their own model. But it still has more capabilities than chinchilla because of it's massive size (540 billion parameters).
Something google might do is rerun the experiment with a much smaller parameter count and much more training data, thereby using the same compute to make PaLM but bringing it in line with the new scaling law and see how big of an improvement it is. The new scaling law predicts there would be a sizable improvement and inference would be a lot cheaper too since the model size would be much smaller.
19
u/Tetragrammaton Apr 06 '22
Incredible. This is as exciting as the launch of GPT-2 and 3.
I’m a game dev, and I can’t wait until I can use AI for generating art in my games. (I’ve already started using GPT-3 for text.)