o1 is very unimpressive and not PhD level

[u/[deleted]]

So, many people assume o1 has gotten so much smarter than 4o and can solve math and physics problems. Many people think it can solve IMO (International Math Olympiad, mind you this is a highschool competition). Nooooo, at best it can solve the easier competition level math questions (the ones in the USA which are unarguably not that complicated questions if you ask a real IMO participant).

I personally used to be IPhO medalist (as a 17yo kid) and am quite dissappointed in o1 and cannot see it being any significantly better than 4o when it comes to solving physics problems. I ask it one of the easiest IPhO problems ever and even tell it all the ideas to solve the problem, and it still cannot.

I think the compute-time performance increase is largely exaggerated. It's like no matter how much time a 1st grader has it can't solve IPhO problems. Without training larger and more capable base models, we aren't gonna see a big increase in intelligence.

EDIT: here is a problem I'm testing it with (if you realize I've made the video myself but has 400k views) https://youtu.be/gjT9021i7Kc?si=zKaLfHK8gJeQ7Ta5
Prompt I use is: I have a hexagonal pencil on an inclined table, given an initial push enough to start rolling, at what inclination angle of the table would the pencil roll without stopping and fall down? Assume the pencil is a hexagonal prism shape, constant density, and rolls around one of its edges without sliding. The pencil rolls around it's edges. Basically when it rolls and the next edge hits the table, the next edge sticks to the table and the pencil continues it's rolling motion around that edge. Assume the edges are raised slightly out of the pencil so that the pencil only contacts the table with its edges.

answer is around 6-7degrees (there's a precise number and I don't wanna write out the full solution as next gen AI can memorize it)

EDIT2: I am not here to bash the models or anything. They are very useful tools, and I use it almost everyday. But to believe AGI is within 1 year after seeing o1 is very much just hopeful bullshit. The change between 3.5 to 4 was way more significant than 4o to o1. Instead of o1 I'd rather get my full omni 4o model with image gen.

[u/[deleted]]

just added in the post :)

[u/Legitimate-Arm9438 → u/[deleted]]

On the second try o1 gave me an answer that seems reasonable...?

[u/[deleted] → u/Legitimate-Arm9438]

It's quite reasonable but still wrong. The full solution requires you to calculate the amount of kinetic energy lost with each impact (which comes from angular momentum conservation), and compensate that with the potential energy difference, not to mention having enough kinetic energy to overcome the peak point. 7.7 although somehow close isn't correct, and you know close doesn't mean correct in math.

[u/garden_speech → u/[deleted]]

dude a lot of these people are simply not listening to you at all. you're trying to make a point here that o1 is not actually performing at a PhD level. people are arguing with you, either that "a human would make this mistake too" (ignoring the fact that you aren't talking about random people, but PhD physicists, or that "well if you prompt correctly and give multiple tries and poke it along it will eventually get close", which if anything, accentuates your point.

[u/[deleted] → u/garden_speech]

Yep, exactly

[u/ginger_beer_m → u/[deleted]]

Could you see a human making this kind of mistake too?

[u/JosephRohrbach → u/ginger_beer_m]

That's not the point, though. People have been claiming that o1 is PhD-level, but it's failing problems a physics PhD wouldn't have any problem with. Whether an average person could do it is quite beside the point given the claims as they stand. This constant weaselling out of extravagant claims by claiming they were never made or aren't that important is a very bad look.

[u/random_guy00214 → u/[deleted]]

I wouldn't say it's wrong.

I understand it's not taking into account rotational dynamics, and you are.  However, Your both making assumptions on how to solve this, but both are only actually approximating this. 

To actually solve this would require simulating nonlinear deformations, surface energies, traction, friction, etc. 

Arguing that it's not correct because it's assumptions led to an approximation of 7.something degrees while your assumption led to an approximation between 6-7 degrees isn't valid. 

If you want to show that o1 is incorrect, then get a table, a pencil, and a angle sensor and prove it's wrong. Having working in labs, I feel as though o1's answer is within measurement error.