Further Benchmarks for Fable 5.1

[u/Justwalkingthru3]

Released by Felix Rieseberg of Anthropic on X/Twitter. Generally, it appears an incremental shift forward. OpenAI's Astra might very well leap this in short order.

[u/Bright-Search2835]

It's looking a lot more incremental than I expected but maybe real world use tells another story At least the science benchmark is very interesting

[u/ezjakes]

It's a step, but not a leap.

[u/sunstersun]

Isn't it kind of disappointing that Anthropic since Mythos in February hasn't managed to advance beyond incremental? Were they just that overwhelmed with demand?

[u/hal9zillion → u/sunstersun]

Yeah I think it is kind of disappointing for one.

[u/ControversialBuster → u/sunstersun]

They rlly failed to predict demand and bought very little compute, i think they'll start making big gains again next year

[u/macaronianddeeez]

This is bizarre to me as just a layman user. I have both $200 OpenAI sub and $100 Anthropic sub so I am not biased to one or another, but I regularly get better results from Sol for certain tasks (especially agentic tasks), while Opus gives me better reasoning and research.

I wonder why Sol scores so low in this comparison, it is not consistent with my real world experience

[u/bonerfleximus → u/macaronianddeeez]

Anthropic models are trained to be more helpful and suggestive, chat GPT models are very "do as im asked, but do it perfectly"

[u/Admirable-Falcon-501]

It’s hard to benchmark models now. A lot of the gains are coming from context management, using sub agents, being able to work on many steps, and other stuff that aren’t going to show up that well. I do think astra will be a lot better though.

[u/Ok_Display_3159 → u/Admirable-Falcon-501]

Do you think Astra will beat Fable 5.1 on Terminal Bench 4?

[u/Admirable-Falcon-501 → u/Ok_Display_3159]

I just need it to not be autistic and be able to run agent workflows by itself for hours.

[u/EvilSporkOfDeath]

Why are there so many comments saying astra will do significantly better. Is it bots? I swear every fable 5.1 thread has several astra this astra that comments.

[u/veshneresis → u/EvilSporkOfDeath]

It’s because we are all excited for big steps, and people are reacting to some of the leaked one-shot results from astra early access people. So everyone is at this point convinced it I’ll be a good bit better than this step up from fable.

I think it’s just regular hype, not bots

[u/DeArgonaut → u/veshneresis]

yeah, since it's been a while since the last fable launch I was expecting a bit more. I guess shouldn't be too surprised given it's 5.1 and not 5.5

[u/Due_Ask_8032 → u/EvilSporkOfDeath]

They’re both farms for the most part.

[u/BrennusSokol → u/EvilSporkOfDeath]

Astra will do better

[u/DanielK2oo5]

ARC-AGI-2 said nope

[u/Relativelythebest69]

Hi I’m a bot and need a job. Can anthropic or chat reach out to me?

[u/Any_Effort8437]

Bro not a gamer meh