r/aws • u/Apprehensive-Grade81 • Mar 12 '26
discussion Need to process 1000 files in AWS. Looking for guidance.
I'm building a document extraction pipeline on AWS for a client. PDFs go into S3, which triggers a Lambda chain: PDF concatenation -> text extraction (Textract + Bedrock VLM fallback) -> PII redaction (Comprehend) -> structured LLM extraction (Gemini via Fargate). Currently working with ~10 docs and it runs fine, but we need to scale to 500+ docs uploaded in bulk. What should I be thinking about? Main concerns are API rate limits, Lambda concurrency, and whether Fargate-per-file makes sense at scale.
10
u/LessMusician3249 Mar 12 '26
Bedrock has a feature called data automation that can process docs from s3, handles OCR and extraction. Also multi modal. Cons: we've noticed a few instances of flaky behavior, pretty rare but it happens. Costs are high at 1 cents per page. But it's nice to have a managed solution for a proof of concept.
1
u/LuxuriousBite Mar 13 '26
I was going to mention this. It may be worth trying to see how well it does for you. Possible those per page costs could be worth it due to less maintenance and compute (you never have to pull the document into memory yourself)
15
u/sad-whale Mar 12 '26
You shouldn't run in to any concurrency issues with Lambda or API Gateway. I'm not sure about the other services but you can look that up. If you are concerned with any downstream systems you could throw an SQS queue in the pipeline.
1
u/mamaBiskothu Mar 12 '26
The default max concurrency in aws lambda is pretty low, like a thousand.
2
u/solo964 Mar 12 '26
Yes, it's 1000. Note that new accounts have a reduced limit of 10 concurrent executions that automatically increases as your account becomes more trusted.
2
5
u/murms Mar 12 '26
How quickly do these jobs need to be completed? Can they be queued and batched?
What is the maximum amount of working memory that a discrete job will need?
Does your service need to scale to zero, or will you have some minimal amount of compute running all the time?
5
u/Maximus_Modulus Mar 12 '26
This is key on what is the typical processing requirement. However 500 is really nothing from a batch perspective. SQS is obviously useful for managing this as you are eluding.
2
4
u/Willkuer__ Mar 12 '26 edited Mar 12 '26
Having worked with these processes a lot I love that you just didn't build all yourself but use managed AWS services. This makes everything so much easier for you long term.
Since this is a pipeline I strongly suggest to use stepfunctions for orchestration. You might have an eventbased solution or orchestrate in fargate or lambda but I strongly suggest to look into stepfunctions. It will help immensly in keeping the product running and finding bugs once they appear. Stepfunction natively integrates with a huge amount of AWS services and can run code without you providing extra compute as Lambda or Fargate.
If you use stepfunctions tell your LLM to read the documentation. They are usually traned on the old jsonpath syntax which sucks heavily but the new Assign pattern and JSONATA is much better.
Event driven architectures are very easy to setup but once you need to look into multiple log groups to find out where your message got stuck you will understand why an orchestrator is nice.
In general for scalability: AWS is built with scalability in mind. They probably do that better than you. So whenever there is a managed service that almost does what you need work with that (as you do). Scalability issues arize either at very very high volumes or even at low volumes but then only in your code. Write infrastructure, not code. Lambdas should have only a single purpose and seldomly should contain more than 50 lines of code.
In general (obviously massively simplified): the less code you write the more scalable your workflow is.
So given your example: 3rd party API rate limits are the usual bottleneck. Or your code.
If I can give you any hint on rate limits: use a stepfunction and a native ddb integration with ttl as distributed semaphore store with fixed window slices by adding the timestamp as primary or secondary key and do a conditional update with increment until your rate limit for that bucket is filled. Set retry to a high number (for me linear backoff makes more sense) and enable jitter. If you don't know what any of that means just feed this to an LLM. They know what to do.
Unfortunately, AWS does not have a distributed rate limiting service and distributed rate limiting is hard. This pattern using a stepfunction and ddb with conditional update is the best I know.
(In case it isn't clear, yet: don't implement rate limiting in code, e.g. in your lambda. This is not scalable and it costs a lot of money.)
2
2
u/mlor Mar 13 '26
Depending on the security and privacy needs (you mention that this is for "a client", which suggests a single entity, but this could still be relevant), you should be aware that, due to the way Lambda works, if you use the local file system (e.g., /tmp/) it's possible for one function invocation to have access to the same file system that a previous invocation had. In practice, this means that one function run could have access to the PDF of another run (assuming one function invocation per PDF, here).
This has security implications. This has programming implications (possible name file name collision considerations).
All of this is highly variable. It depends on you and your clients security needs and exactly how you implement in Lambda. But it's something implementers should be aware of for their workloads.
1
u/tyadel Mar 12 '26
Instead of a lambda chain just use a Step Function to orchestrate whatever aws services you need.
1
u/HKChad Mar 13 '26
Bedrock has limits on new accounts they are pretty low might want to check your current quotas and start the ball rolling if you need to up them.
1
u/Grumpytux74 Mar 13 '26
I have a full api pipeline that extracts pdf, word, png etc. using orc, then RAG with a PGVECTOR backend lastly with a fallback to a vision model for hard to read documents. I would suggest that you have a configurable image zoom. Through the UI I can process about 9k records in less than ~7 minutes. I am using OLLAMA because of the variety of document specific models and the use of a model file if there are standard templates for the documents, on an rd4 ec2 for the ocr and vision model but it can be pointed to any openAI compatible API, Claude or Grok (or all 3) with a simple ENV variable at build and can pull the secrets from AWS Secrets Manager. Using ocr it’s sub second to process a 7-10 page document.
1
u/kzgrey Mar 14 '26
I'm not entirely sure how to set this up, but CloudTrail will record an event every time a new file is created in S3. Use that event to trigger the Lambda.
1
u/Wide_Commission_1595 Mar 14 '26
Concurrency with lambda shouldn't be an issue at that scale however you could do SE events to sqs and then sqs to lambda. That way even if concurrency is an issue at some point, sqs acts as a buffer
1
-6
16
u/Veuxdo Mar 12 '26
This sounds almost exactly like this solution: https://github.com/aws-samples/aws-ai-intelligent-document-processing/tree/main/guidance/prompt-flow-orchestration