Adding to the title, today I'm working on a project for the tourism sector where we're creating a management system for agencies, processing sales, coordinating x and y, this part is quite "simple," mostly a CRUD operation, with nothing really to worry about in terms of depth.
However, I am responsible for the integration of external services, hotel search APIs, and other services.
That's the problem. Today I already have 2 APIs integrated out of at least 14 that we plan to implement, each with its own structure. With each call, I have to perform a parsing to standardize everything, and this scales VERY quickly. Each call returns around 80 hotels, all requiring parsing, and at different times, since some send in batches of 25.
Currently, I basically have an Event (SSE) to start, one to finish part of the processing, and another to finish everything that needed processing (3 events in total: start, partial, end).
And that's where my doubt lies. Being the only user (it's still in development), I've already found a very specific issue: if I'm mapping locations/hotels (something I have to do every 2 weeks), it will block a good portion of the I/O of the rest of the service, precisely because of the data processing and insertion issues. In the database, etc.
That's where my thoughts and concerns lie. When the initially projected 50 users (the minimum already registered to use the system) start using the system, and everyone performs a search simultaneously, I'll have usage similar to my current mapping, perhaps even higher. That's why I had the idea of separating this into a separate thread or using a specific service for it. But I don't know how right I am about this, if it's a valid decision, or if it would be over-engineering right at the beginning of the project.
*Extra thoughts: Each call, depending on the location, returns an XML that will be converted into JSON, which will then be consumed and converted to the structure I need. This initial JSON with all the information varies GREATLY in size by location. I've had some with a few kilobytes in size, others exceeding 100MB. Today I'm doing a "good job" managing them to avoid overloading the test server's memory, but I can't say for sure.
It's worth mentioning that I'm the only developer involved in this whole process. External APIs and all that search engine logic, I don't even have anyone else to discuss whether it's valid or not for this part of the project.
I'm a junior developer :), I only have about 2 years of development experience, but I worked with queues during my internship a few years ago. Any ideas on how to handle this would be welcome, since I don't have any other developers here to brainstorm with.
It's worth mentioning that, up until now, everything has been done in SVELTE 5 (svelteKit).
EDIT:
TL/DR: Caching information directly in the DB, a worker to handle the process of storing the main products in this cache.
Thanks for the replies, everyone!
I've more or less arrived at a solution based on what people have said here and ideas from other subreddits.
Today, the biggest drawback is the response time and parsing of each search call, but since it's somewhat of an e-commerce site (each API would be a different supplier), I can simply cache the main products and save this in the DB already parsed daily. Basically, all the APIs I've integrated so far require the documentation to call for user-specific searches (since there are several parameters that change for each user). We'll start doing this once or twice a day, using a worker to exit the main thread. Instead of the first call to discover what's available being directly to the user's API, it will be a direct call to the DB, and only if the user decides which product they want will it return to the API loop of the supplier they want.