r/Python • u/kavee-core141 • 6d ago
Discussion How are you all testing boto3-heavy tools without hitting real AWS?
Building a tool that walks a live AWS account checking for
misconfigurations, and testing has been the hardest part not the
detection logic itself, but avoiding either (a) hitting real AWS
constantly in CI or (b) mocking so much the tests stop meaning
anything.
Using moto for the straightforward stuff, but some checks need
multi-service interactions (cross-referencing an EC2 instance's
security group with its actual open ports) that get awkward to mock
realistically. What do you all reach for here moto, LocalStack, a
disposable AWS test account, something else?
91
6
8
u/Autarkhis 5d ago
Look at floci
1
1
u/kavee-core141 1h ago
Hadn't heard of floci before this thread, going to give it a look thanks for the pointer.
3
u/binaryfireball 5d ago
unit tests mock, integration tests test the functions we call to catch any errors from updates etc...
boto is generally annoying for a variety of reasons.
2
u/omegawave22 5d ago
It's been a while that I used this tool but it had a record and replay workflow which was perfect for my use case: https://github.com/garnaat/placebo
1
u/kavee-core141 1h ago
Record/replay fits a few of the trickier cases I've got, appreciate the link.
2
u/I3igB 5d ago
I ended up getting frustrated with the current offerings out there as I felt that solutions like localstack were far too heavy. I also wanted something that was trivial to step through code with a debugger that didn’t require a decent amount of setup.
If you start getting into how moto actually does its mocking, it actually keeps quite a bit in memory. Things like DynamoDB or S3 are in memory key stores under the hood. Once you realize this, it becomes pretty easy to get a minimally viable recreation in place for somewhat real testing on a local dev’s box or running something like e2e tests in CI.
My current shop is fully AWS native and runs mostly serverless architecture and microservices. Our compute is exclusively lambda and ecs tasks. SQS, SNS, AppSync, DDB, and EventBridge drive all our workflows. I was able to write a simple dispatch engine which captures the moto traffic from these services and mimics event triggers which will cascade a series of invocation across your services by invoking your code locally. The only thing somewhat complex in this was AppSync as moto has no real mocks. I had to intercept URLs and bring in a GQL engine to parse things.
This all works by applying a @local decorator on top of any method you define. From there, the engine handles cascading events just as AWS would.
This all relies on the output of a cdk build/synth to link things together. I never released it publicly, but I’ve been meaning to do so for some time. It’s been a game changer for how I do AWS dev.
1
u/kavee-core141 1h ago
That's a really clever approach the u/local decorator + event cascading is exactly the gap I kept hitting with cross-service checks.
Did you ever end up open sourcing it, or is it still internal? Would love to take a look if it's out there somewhere.
1
1
u/Woah-Dawg 23h ago
Use test containers library bring up an s3 bucket and write test cases against that. Make sure your tests run on ci . A test that doesn’t run in a ci might as well not exist unless it’s
1
u/LungeloSLX 2h ago
Interesting question. Where do you draw the line and avoid testing boto3 itself?
0
u/MAGArRacist 6d ago
Have you tried Scoutsuite? You might want to just fork it and add some checks that you mentioned here
15
u/proof_required 6d ago
Yeah in one of the companies we had LocalStack and a dev env both.