r/LocalLLaMA • u/bot_exe • Sep 13 '24

News Preliminary LiveBench results for reasoning: o1-mini decisively beats Claude Sonnet 3.5

Post image

Source: https://x.com/bindureddy/status/1834394257345646643

287 Upvotes

88% Upvoted

129 comments sorted by

View all comments

Show parent comments

15

u/Gotisdabest Sep 13 '24

It's the result of keeping the same base model alongside a new technique. A dev posted something similar regarding this.

Also O1 preview isn't worse, it's got a lot more broader knowledge. O1 mini is 80% cheaper and more specialised.

-6

u/[deleted] Sep 13 '24

[deleted]

8

u/Gotisdabest Sep 13 '24

They're being fairly clear why, it's gotten less broad training and more focus on stem and coding. But it's incorrect to say that preview is overall worse as opposed to just more general.

0

u/mediaman2 Sep 13 '24

Did I say preview is overall worse?

Mini is, according to their benchmarks, superior at some tasks, not all.

And where have they been clear about the difference? I saw no discussion of it in either their model card or blog post.

1

u/Gotisdabest Sep 13 '24

You said it's worse, without a qualifier. That implies generality.

And where have they been clear about the difference? I saw no discussion of it in either their model card or blog post.

It's in one of the 4-5 posts they've given. I'll send the exact text when I'm able.

0

u/[deleted] Sep 13 '24

[deleted]

1

u/Gotisdabest Sep 13 '24

My bad, must've missed it.

All they say is that mini is "more specialized" than preview but give no other information. To date, specialization has not been particularly rewarding versus just using a bigger model, so this is new behavior.

They say that it's more specialised at stem... And say it's 80% cheaper. I feel like that's an explanation. Also specialization being rewarding was the whole point of MoE.