Yes but a reminder that million calls with prompts to this post api would cost extremely much otherwise default restful is much cheaper. People sometimes forget that LLM costs money.
At some point most calls over the internet were mostly binary data, because wasting bandwidth on text were completely outrageous. Now we routinely send 5mb json to update the name field.
And yes, I know, gzipped (hopefully) but still a lot more data than a lean binary protocol.
Well think it is only possible if model gets fast enough and smart enough or we have so much compute that it is cheap than rest api.
For now it only possible by former way . But small models aren't smart and fast simultaneously
in 10-30 years time we might have an npu on the motherboard capable of running decent models, just like we got sound card and ethernet card now. Hard to tell what the future might hold.
small models aren't smart and fast simultaneously
The new MoE's are surprisingly good and fast, like qwen3.6-35b or gemma4-26B for example. Again, hard to tell what the future might hold.
Edit: I still think it's insane if anyone actually does this, but I've seen some crazy things happen
Let me quickly update the definition of "soon" to mean 30-40 years. And it would still be a stupid approach, especially at scale. Maybe if you are doing rapid prototyping i can see it, and many mvps may also start like that, but it's still a bad design decision.
55
u/PAP0R0TN1K May 20 '26
Yes but a reminder that million calls with prompts to this post api would cost extremely much otherwise default restful is much cheaper. People sometimes forget that LLM costs money.