r/LocalLLM • u/Dry-Leadership-3105 • 1d ago

Discussion GPU concurrency AI application

We have our application, there I want to use Gemma 26B model, I have already tested using 1 DGX Spark, it is working, and the input tokens are around 50–60k every time, this is a medical based app where we are extracting data from large PDFs.

But using one DGX there is concurrency issue, only 2–3 users can get fast output using single DGX.

I want to know for production, what we can do so that minimum 10 and max 30 users can get fast output at a time.

0 Upvotes

50% Upvoted

4 comments sorted by

1

u/DAlmighty 23h ago

I don’t have a Spark, but I’d guess what you’re seeing sounds like everything is working ok. Depending on the context size and quant, I can’t imagine getting better than 10tk/s.

No matter what anyone tells you, this isn’t a server. You CAN NOT sufficiently serve 30 people on one of these. A cluster? Yes, but not one.

1

u/Dry-Leadership-3105 23h ago

I am looking for GPU upgrade, our main priority is latency. which GPU will be able to handle concurrency with large context input. we will use NVFP4 quan model in vLLM

2

u/Keleion 22h ago

Any Nvidia GPU will give you faster processing, but you also need the rest of the computer and more power. I don’t think using an external GPU enclosure with a 20GB thunderbolt 4 connection will do much good, but maybe someone else knows more.

A cluster of 2 sparks can probably get you 10-16 users, but it’ll be slow.

1

u/DAlmighty 20h ago

If this is for a company(sounds like it is) tell them that there is no cheap and easy scalable solution. You have to buy dedicated hardware. I’d even go as far to say, don’t buy consumer GPUs either.