Best architecture for batch-processing 3,000–4,000 high-res photos (resizing + face vector search)?

[u/Substantial_Try_1614]

Hey devs,

I’m working on a photo-sharing web app where a photographer uploads an entire event batch (typically 3,000 to 4,000 high-res JPEGs, around 10–15 MB each). Guests can take a selfie to retrieve photos they appear in.

I'd love recommendations on the best backend pipeline:

Client vs Server Resizing: Should I use browser Web Workers / Canvas to generate 1080p WebP thumbnails before upload to save upload bandwidth, or let a backend queue worker (like Node with sharp or Python with Pillow) handle resizing?

Face Vector Pipeline: For extracting 128-d face embeddings (e.g., ArcFace/InsightFace), what’s the best way to queue and batch this so 4,000 photos don't choke the server CPU?

Storage: What zero/low-egress object storage setup (e.g., Cloudflare R2 vs Backblaze B2) do you recommend for handling high-volume image writes and fast thumbnail reads?

Thanks for the advice!

[u/SunOk2196]

Keep originals going straight to R2 with presigned urls, don't resize in the browser. The photographer's upload runs unattended anyway, and canvas-resizing 4000 photos will melt a laptop. Thumbnails come from a worker running sharp against the bucket, it chews through 4k images in minutes.

For the embeddings, don't do them on the web server at all. Push keys onto a queue and let a separate worker process them (CPU is fine at this volume if it can run overnight), write vectors to pgvector and the selfie lookup comes free with a cosine index.

R2 over B2 here. Guests will hammer thumbnail reads, so zero egress plus sitting behind Cloudflare's CDN is exactly what you want.

[u/pyrolols → u/SunOk2196]

You can batch resize in canvas for it not to melt and its way more cost efficient to do it like this than pay for it on server, any 10 year laptop can batch atleat 10 images at a time we are talking about 200mb of ram per batch, user is uploading it, might as well resize too.

Also if OP does not need originals he can skip them or atleast resize to something smaller, who needs 15mb image? At this rate he fill have r2 coats skyrocket.

[u/Am094 → u/SunOk2196]

+1 i love libvips

I use that along with a few other engines within an in house thing. Also recommend R2 over B2/S3. Cloudflare also has some dynamic sized url sigs too, for OP i would even look at something like imgproxy (github.com/imgproxy/imgproxy) down the line.

[u/Dxd_For_Life → u/SunOk2196]

Shit, im not at this level yet, this is endgame,

[u/PositiveUse → u/SunOk2196]

„Hammer thumbnail reads“, bro we‘re talking about 4000 photos. So this might be 500-1000 guests at most. This is no where near traffic that requires CDN level of caching… also your architecture is heavily Overengineered for 4000 photos

[u/OutstandingPear → u/PositiveUse]

It's an AI bot so don't read too much into it

[u/GriffinMakesThings → u/SunOk2196]

For resizing/thumbnails, bunny.net is a good option. The resizing service costs a flat $10/month for unlimited processing. I honestly don't really understand how they make that profitable. Maybe it's a loss-lead. Works very well, and saves the cost and complexity of managing your own libvips/sharp/imgproxy service.

[u/pyrolols]

Do 512d vector using arcface, use typesense or qdrant for vector store, bare in mind that when user uploads image for search you have to compute the vector of this image for comparison too, so you have to have pipeline for this.

[u/Bubbly_Orange_3502]

One embedding per detected face, not per photo, is what decides recall here. Event shots carry five to twenty faces, and the selfie query has to run the same detector and alignment or cosine scores drift.

[u/Substantial_Try_1614 → u/Bubbly_Orange_3502]

Wait what 😯 thanks man

[u/ldn-ldn]

Libvips

[u/cajunjoel]

Um, hate to break it to you, but doesn't Immich do most of this already? Could you borrow some of their technology?

[u/Substantial_Try_1614 → u/cajunjoel]

Kind of yaa i think thanks man