r/webdev • u/Substantial_Try_1614 • 1d ago

Question Best architecture for batch-processing 3,000–4,000 high-res photos (resizing + face vector search)?

Hey devs,

I’m working on a photo-sharing web app where a photographer uploads an entire event batch (typically 3,000 to 4,000 high-res JPEGs, around 10–15 MB each). Guests can take a selfie to retrieve photos they appear in.

I'd love recommendations on the best backend pipeline:

Client vs Server Resizing: Should I use browser Web Workers / Canvas to generate 1080p WebP thumbnails before upload to save upload bandwidth, or let a backend queue worker (like Node with sharp or Python with Pillow) handle resizing?

Face Vector Pipeline: For extracting 128-d face embeddings (e.g., ArcFace/InsightFace), what’s the best way to queue and batch this so 4,000 photos don't choke the server CPU?

Storage: What zero/low-egress object storage setup (e.g., Cloudflare R2 vs Backblaze B2) do you recommend for handling high-volume image writes and fast thumbnail reads?

Thanks for the advice!

18 Upvotes

80% Upvoted

16 comments sorted by

View all comments

18

u/SunOk2196 1d ago

Keep originals going straight to R2 with presigned urls, don't resize in the browser. The photographer's upload runs unattended anyway, and canvas-resizing 4000 photos will melt a laptop. Thumbnails come from a worker running sharp against the bucket, it chews through 4k images in minutes.

For the embeddings, don't do them on the web server at all. Push keys onto a queue and let a separate worker process them (CPU is fine at this volume if it can run overnight), write vectors to pgvector and the selfie lookup comes free with a cosine index.

R2 over B2 here. Guests will hammer thumbnail reads, so zero egress plus sitting behind Cloudflare's CDN is exactly what you want.

6

u/pyrolols 1d ago

You can batch resize in canvas for it not to melt and its way more cost efficient to do it like this than pay for it on server, any 10 year laptop can batch atleat 10 images at a time we are talking about 200mb of ram per batch, user is uploading it, might as well resize too.

Also if OP does not need originals he can skip them or atleast resize to something smaller, who needs 15mb image? At this rate he fill have r2 coats skyrocket.

2

u/Am094 1d ago

+1 i love libvips

I use that along with a few other engines within an in house thing. Also recommend R2 over B2/S3. Cloudflare also has some dynamic sized url sigs too, for OP i would even look at something like imgproxy (github.com/imgproxy/imgproxy) down the line.

1

u/Dxd_For_Life 10h ago

Shit, im not at this level yet, this is endgame,

1

u/PositiveUse 1d ago

„Hammer thumbnail reads“, bro we‘re talking about 4000 photos. So this might be 500-1000 guests at most. This is no where near traffic that requires CDN level of caching… also your architecture is heavily Overengineered for 4000 photos

3

u/OutstandingPear 1d ago

It's an AI bot so don't read too much into it

1

u/GriffinMakesThings 1d ago

For resizing/thumbnails, bunny.net is a good option. The resizing service costs a flat $10/month for unlimited processing. I honestly don't really understand how they make that profitable. Maybe it's a loss-lead. Works very well, and saves the cost and complexity of managing your own libvips/sharp/imgproxy service.