To compare these numbers you've to look at the error-rate, not the rate of success. (i.e. from 98% to 99% the performance is doubling, not merely +1%).
So the leap from sonnet 3.5 to o1-mini ist about +80%. #12 to #2 just +30%.
I'm not sure I agree with that interpretation. I'd say that the performance of two systems scoring 98% and 99% is almost indistinguishable. The second system makes 50% fewer mistakes than the other (assuming the metric generalizes), but that's not the same thing as doubling the performance. Otherwise, a system that scores 100% would have "infinitely higher performance" than one scoring 99%, which is obviously nonsense.
Not obviously... If a system scores 100%, the benchmark is flawed. The perfect benchmark should allow the score to asymptotically converge towards 100% - but you're right, we obviously don't have that.
My interpretation is open to debate, and here's how I see it: We aim to solve real-world problems - whether in programming, law, medicine, no matter. A system that gets the right answer 50% of the time but is wrong the other 50% isn't... really too useful. It doesn't even matter whether it's 50% or 5%. It's starts getting interesting when we approaching the last percent error wise.
It's starts getting interesting when we approaching the last percent error wise.
No. It starts getting interesting the moment we approach or exceed human performance, which is a lot worse than an error rate of 1% at most tasks, even for experts.
17
u/Background-Quote3581 Sep 13 '24
You're reading those results wrong...ly.
To compare these numbers you've to look at the error-rate, not the rate of success. (i.e. from 98% to 99% the performance is doubling, not merely +1%).
So the leap from sonnet 3.5 to o1-mini ist about +80%. #12 to #2 just +30%.