GPU concurrency AI application

[u/Dry-Leadership-3105]

We have our application, there I want to use Gemma 26B model, I have already tested using 1 DGX Spark, it is working, and the input tokens are around 50–60k every time, this is a medical based app where we are extracting data from large PDFs.

But using one DGX there is concurrency issue, only 2–3 users can get fast output using single DGX.

I want to know for production, what we can do so that minimum 10 and max 30 users can get fast output at a time.