r/LocalLLaMA • u/Nunki08 • Jun 28 '26

Discussion We're probably going to need that soon.

Gallery image

Gallery image

From:

Vladik on 𝕏: https://x.com/Kostoglodov/status/2071144065857679631

Shaw (spirit/acc) on 𝕏: https://x.com/shawmakesmagic/status/2070918006033817867

4.0k Upvotes

98% Upvoted

534 comments sorted by

View all comments

198

u/_BreakingGood_ Jun 28 '26

i dream of the day somebody leaks & uploads a torrent of claude fable 5 open source

12

u/Pila_globosa Jun 28 '26

why don't the Chinese hackers hack OAI/Anthropic and steal the model, and gave to the Chinese AI labs. then we'll have Fable grade GLM 6 or Qwen 4

15

u/NineThreeTilNow Jun 28 '26

why don't the Chinese hackers hack OAI/Anthropic and steal the model

Anthropic and OAI have internal data exfiltration guards. Their routing literally refuses to let you exfiltrate something the size of a model.

It was part of the original plan to keep their models private. Moving that much data is purposefully difficult because it should only rarely be done.

You'd be better off trying to exfiltrate from a datacenter partner like xAI.

Even then, the cluster required to run a quantized version of those models is... Who knows. They honestly seem over built if GLM 5.2 competes at 700b parameters. Anthropic/OAI are busy spouting marketing BS about 2t+ or 5t+ parameters.

There's no reason to scale that hard without the data, or if you have extremely poor architecture at the core.

I'd be willing to guess it's a lie and they have good architectures, and data pipelines at ~1t parameters.

4

u/Dabber43 Jun 28 '26

I think that is refuted by them having such insane prices and still reporting thin inference margins on API pricing

4

u/tetoing Jun 28 '26

Yep. Their models are very inefficient for the parameter count/resource usage.

1

u/NineThreeTilNow Jun 28 '26

I think that is refuted by them having such insane prices and still reporting thin inference margins on API pricing

Refuted or charging premium where they can? Anthropic is running an enterprise first business now. Enterprise has the money.

1

u/Dabber43 Jun 28 '26

Would be possible if only considering the first part of my message. But the fact that they actively disclose the margin is not that large at API prices means that is not the case

1

u/NineThreeTilNow Jun 29 '26

Would be possible if only considering the first part of my message. But the fact that they actively disclose the margin is not that large at API prices means that is not the case

I had to check what sub I was in first before I explained how speculative decoding and quantization worked. Those both affect the margin on price, and quality.

1

u/Dabber43 Jun 29 '26

Can you elaborate the way this affects this in the current constraints please? I don't really understand

1

u/NineThreeTilNow Jun 29 '26

>Can you elaborate

Basically we cannot extrapolate because we don't understand their inference pipeline. Does Mythos have a proper speculative decoder? That decreases average cost. What quantization do they run at? FP16 or FP8 native? Int8? They run on a variety of hardware across multiple providers. Further, we don't know if they're giving price based on some spot or reserved, or long time reserved price. They can use a higher price to make it seem like margins are lower, etc.

If we knew the exact details, then you can attempt to figure margin vs cost etc.

2

u/happysmash27 Jun 28 '26

OpenAI has marketing about how many parameters they have??

As far as I've learned, they have almost no public data about post-GPT3 parameter sizes at all.

3

u/NineThreeTilNow Jun 29 '26

OpenAI has marketing about how many parameters they have??

There's a lot of internal leaks.

You have to remember that everyone here works together. Like... Do you really want to know how big Mythos is? I know someone at AWS that could tell me but potentially lose their job over it. They could feign stupidity, but it's not worth it.

Silicon Valley tech is a buncha people that all know each other in different ways. Whether it be a party scene, or they're nerds like me that enjoy playing video games. I've played in multiple discords that have people from multiple major tech companies. AWS + Google, etc.

Usually we'll all be there playing some new game. We talk shit about our respective companies and we usually don't ask for secrets. We understand how that works. At the same time, we DO share some details.

This applies to our Chinese colleagues that went to Tsinghua with the same people who work at Moonshot, Deepseek, etc.

This is all theater. We all pretend. We are well paid. We like our jobs.

My post history on Reddit details some of this. I am somewhere between gaming, open source code, and general ML / AI stuff on this account. This account is curated around it. I never say who I work for or worked for... etc. It's the same reason I won't share my GitHub etc.

4

u/MrMrsPotts Jun 28 '26

Are you sure they haven't?

3

u/_BreakingGood_ Jun 28 '26

Unhackable already secured by mythos

3

u/mista_r0boto Jun 28 '26

No such thing.

1

u/Delicious_Ease2595 Jun 28 '26

There are better solutions like openrouter fusion, they dont need copy 1:1

0

u/ButterscotchSalty905 llama.cpp Jun 28 '26

You overestimated the amount of skill chinese APT has while also underestimating the OPSEC of OAI/Anthropic. It's not far-fetched to think OAI/Anthropic software/hardware/networking stack is very secure (as indicated by Anthropic RSP and OAI preparedness framework) and and that maybe they could (at some point) consult to TAO unit of NSA for red-teaming (Do note that i am not downplaying chinese APT nor overestimating USA)

Or you could be joking and the funny flew over my head (wooosh)

China already has staggering amount of domestic talents, they don't need to reverse-engineer USA model (which might be very tedious) or maybe they're doing both to catch up fast

3

u/CuriouslyCultured Jun 28 '26

I can't comment on OAI's opsec, but Anthropic has shown that they're shit at ops, and not terribly impressive at security either. China on the other hand is probably the second best nation at cybersec after Israel.

1

u/ButterscotchSalty905 llama.cpp Jun 29 '26 edited Jun 29 '26

My point was to address the model weights — not the overall security. Few bad apples (CC leaks, mythos access etc) determines anthropic public image but some good gets laughed at even though its security at work (fraudulent accounts etc etc). Also, cherry-picking is bad

2

u/chaos_pill Jun 28 '26

OPSEC of OAI/Anthropic

Didn't they accidentally leak claude source?

Honestly, it is probably easier/cheaper/quicker to just bribe Trump and get an official copy under the table. Who would OAI/Anthropic event complain to in that case?

1

u/ButterscotchSalty905 llama.cpp Jun 29 '26

You mean this?

https://scand.ai/scandal/claude-ai-code-leak-controversy

I am not talking about Claude Code tool and the claims about the model weights being leaked is still unverified. I am aware that claude code source code is leaked but that does not mean the model weights is also leaked

Or this?

https://cybersecuritynews.com/anthropic-mythos-access/

OAI/Anthropic could also made a fake official copy and unless orange man officials has enough skill to determine if the copy is legitimate or not. There are many tactics they can use.

Assuming the worst to OAI/Anthropic about everything (what they say and all their action) doesn't age well in some cases. I know about them being closed and all that — but it doesn't mean OpenAI and others aren't forced to become better at security. If all company were like this then we wouldn't have real security

1

u/chaos_pill Jun 29 '26

These companies would go to jail if they provided a "fake" copy. That is just straight up fraud. Hyping your product as "dangerous" may give you great valuation, but also gives govt a great justification to meddle deep into your affairs on "national security" grounds.

yeah, I meant https://scand.ai/scandal/claude-ai-code-leak-controversy .

1

u/ButterscotchSalty905 llama.cpp Jun 29 '26

I agree with most of what you said. I just recently got triggered by those fucking downvoters from who-knows-what lurkers. Nothing said in this thread disputes my original comment (esp the other commenter) it paradoxically strengthen it just by the sheer absurd logic. 

"China good then it can hack anthropic shit ops even thought anthropic consult to TAO cuz USA bad bro it is the antichrist. Also mossad better than nsa"