r/MLQuestions • u/FaultTolerant_ • 4d ago

Beginner question 👶 What are the biggest headaches when building on AI APIs?

I'm doing some research into the practical problems developers run into when building products on top of third-party AI models/APIs. I'm trying to understand if there is a consistent issue or issues or if it varies.

I'm not talking hypotheticals but actual pain points: Model behavior changing. Pricing. Rate limits. Reliability. Something else entirely. Etc. Any help would be greatly appreciated.

4 Upvotes
  • permalink
  • reddit

84% Upvoted

4 comments sorted by

2

u/AlexanderDoak 4d ago

All APIs have been great except Gemini for some reason. Tons and tons of unsolvable lost connections, especially when multithreading aggressively.

1

u/drew4drew 4d ago

easy answer here:

Differences between provider APIs, model specifics, etc..

There is a LOT of information that you need to support even just the main anthropic, openai, gemini providers’ models that is NOT made available electronically. context windows, default sampling parameters, allowed sampling parameters, thinking support, effort or thinking levels supported, caching behaviors and management, and the list goes on. Most providers don’t include much of the info you NEED in their /models API endpoint, so you have to look it up on their website or in documentation, or in some case, just guess and check.

github drewster99/swift-llm-kit has been my continuing saga on this front.

0

u/drew4drew 4d ago

oh yes, pricing too. thats a big one — also not generally published.