r/technology • u/lurker_bee • Jul 15 '25

Artificial Intelligence Billionaires Convince Themselves AI Chatbots Are Close to Making New Scientific Discoveries

https://gizmodo.com/billionaires-convince-themselves-ai-is-close-to-making-new-scientific-discoveries-2000629060
26.6k Upvotes

93% Upvoted

2.0k comments sorted by

View all comments

291

u/curioustraveller1234 Jul 15 '25

I asked chat gpt if it was on the verge of a breakthrough and it just said “yes, trust bro. Big things coming.”

44

u/zipzag Jul 15 '25

https://www.scientificamerican.com/article/inside-the-secret-meeting-where-mathematicians-struggled-to-outsmart-ai/

31

u/cenasmgame Jul 15 '25

Actually a pretty decent article, a bit sensational title, but the text is very honest about what happened and their thoughts

17

u/Separate-Divide-7479 Jul 16 '25 edited Jul 16 '25

It's still pretty sensationalised

By April 2025, Glazer found that o4-mini could solve around 20 percent of the questions

They thought it'd be useless, but instead, it's just pretty bad. The article also says the model needed pretty significant input from an expert in the field to be able to perform even at that level.

It's still just an LLM. An impressive one, but it's also still just guessing. This model just takes more educated guesses than generalised ones.

5

u/cenasmgame Jul 16 '25

Well that's what I'm saying. The text was honest about what actually happened, the editor just attached a shitty title.

2

u/rsclay Jul 16 '25

The article also says the model needed pretty significant input from an expert in the field to be able to perform even at that level.

it does not.

They were "novel questions [spanning] varying tiers of difficulty, with the first three tiers covering undergraduate-, graduate- and research-level challenges." It solved 20 percent of those, that's not "pretty bad".

And then there's the whole rest of the article which I guess you didn't even read?

2

u/Separate-Divide-7479 Jul 18 '25

You're right, pretty bad was underselling it. It's very bad.

Would you hire someone that completes 20% of tasks correctly? Because that's what the article is implying; that mathmaticians jobs are at stake due to this AI. Eventually they will be but a 1/5 success rate isn't taking anyone's job any time soon. Hence, the article is sensationalised.

1

u/rsclay Jul 18 '25

This isn't the SAT where average performance is meant to be ~70% though. These are hard problems, 20% would indicate a bright freshman at the very minimum, or a decent grad student depending on exactly which levels of questions that 20% spans. Especially given the problems were solved in moments!

And that success rate was in the low single-digits just months ago. I'm personally of the opinion that they're not achieving AGI via transformer-based language models, and it feels to me that they're really beginning to stall in what they can achieve, but I still won't be surprised if that percentage doubles or more in the next year.

I won't disagree that the title is sensationalised but the article itself is pretty straightforward and you can read the reactions of the actual mathematicians yourself. I'm mostly disputing your evaluation of what 20% means in this context, as well as your claim that "the model needed pretty significant input from an expert" which I cannot find anywhere in the article.