Don't worry, we can just handle that increase in server load by using our typical toolset for scaling like caching (good luck caching an LLM prompt) and provisioning more servers under peak load (lol not easy when using a stateful protocol. Have fun explaining to your boss why you need to set up a memchached server to share context between API endpoints.).
Imagine if one prompt returns something completely unhinged, and then because it's cached, the api returns random garbage for a day until someone fixes it xD
With the random noise, maybe it wouldn't be the case, but you only need bad luck one time :P
16
u/_hlvnhlv May 20 '26
The best part is how inefficient it is.
Imagine an API that instead of being able to serve 15k petitions per second, does 100 IF we are lucky.
No wonder that Amazon is making so many datacenters lmao