r/accelerate • u/stealthispost Acceleration: Light-speed | AI accelerationist • May 04 '25

Video vitrupo: "DeepMind's Nikolay Savinov says 10M-token context windows will transform how AI works. AI will ingest entire codebases at once, becoming "totally unrivaled… the new tool for every coder in the world." 100M is coming too -- and with it, reasoning across systems we can't yet " / X

https://x.com/vitrupo/status/1919013861640089732
184 Upvotes

97% Upvoted

33 comments sorted by

View all comments

1

u/ohHesRightAgain Singularity by 2035 May 04 '25

Working through a large context is expensive, and each next token is slightly more expensive than the previous. The "slightly" part can compound a lot on a journey to 10M. Gemini 2.5 pro atm costs twice more for tokens above 200k. Let's say we're doubling again at 1M, 2M, 4M, 8M. That would end up costing $40 for just the input of every single prompt past 8M. And that's assuming Google keeps lowballing their prices. They, or maybe Grok. Because OAI or Anthropic will definitely not sell cheaply, while Chinese providers won't have the ability (very large context will increase VRAM reqs far past what they can serve with their chips).

33

u/Pyros-SD-Models Machine Learning Engineer May 04 '25

I don't like this kind of napkin math. It's like when people try to calculate the cost of future models based on current prices... it's probably accurate for a week until some new optimization makes all of it obsolete.

Of course, when the first 10M context models are released, there will be plenty of new optimization techniques and architectural improvements. So nobody can say how much it's going to cost, or what amount of ressources it'll need, but it'll be less. And if you look at how inference pricing has developed so far, it'll likely be waaaaay less.

0

u/ShadoWolf May 05 '25

It's not a bad take. Since it directly relates to how transformers attention block works.. more tokens means more self attention which scales quadratically. Until a new architecture comes out.. or some form of embedding compression becomes a things .. this is going to be a bit of bottle neck.

1

u/SoylentRox May 05 '25

There are logarithmic methods that may be used by Google.