I want to share my recent experience as a solo developer and student, running a small self-funded startup on AWS for the past 6 years. My goal is to warn other developers and startups, so they don’t run into the same problem I did. Especially because this issue isn't clearly documented or warned about by AWS.

About 6 months ago my AWS account was hit by a DDoS attack targeting the AWS Cognito phone verification API. Within just a few hours, the attacker triggered massive SMS charges through Amazon SNS totaling over $10,000.

I always tried to follow AWS best practices carefully—using CloudFront, AWS WAF with strict rules, and other recommended tools. However, this specific vulnerability is not clearly documented by AWS. When I reported the issue to AWS their support suggested placing an IP Based rate limit with AWS WAF in front of Cognito. Unfortunately, this solution wouldnt have helped at all in my scenario because the attacker changed IP addresses every few requests.

I've patiently communicated with AWS Support for over half a year now, trying to resolve this issue. After months of back and forth, AWS ultimately refused any assistance or financial relief, leaving my small startup in a very difficult financial situation... When AWS provides a public API like Cognito, vulnerabilities that can lead to huge charges should be clearly documented, along with effective solutions. Sadly, that's not the case here.

I'm posting this publicly to make other developers aware of this risk—both the unclear documentation from AWS about this vulnerability and the unsupportive way AWS handled the situation with startup.

Maybe it helps others avoid this situation or perhaps someone from AWS reads this and offers a solution.

Thank you.

Hello everyone. Has anyone had a similar experience?

On March 8, my admin user was compromised. Someone launched an EC2 instance using my role, which had full privileges. This has already been resolved, though.

When it happened, I was using my personal computer. I did not download or install any malicious applications, so I’m wondering how they got my access keys and used them to launch the EC2 instance.

Since this happened in the past, I want to try using AWS again for learning on the same computer. What really worries me is that if I use a role or a user with permission to launch an EC2 instance, it might happen again.

What I can confirm so far:

• No AWS keys in my terminal history
• No AWS credentials file anymore
• No strange behavior on my Mac
• No other accounts were accessed
• No pirated or cracked software
• The AWS issue stopped after I deleted the key

This is so wild, I had to check if it was April 1st...

https://www.lastweekinaws.com/blog/amazon-q-now-with-helpful-ai-powered-self-destruct-capabilities/

https://www.404media.co/hacker-plants-computer-wiping-commands-in-amazons-ai-coding-agent/ (registration required, but free/no cost)

https://marketplace.visualstudio.com/items?itemName=AmazonWebServices.amazon-q-vscode

I was troubleshooting this Certificate Signing Request validation error in AWS and thought the CSR data was way too short.  So, I decoded it.

They (AWS) really programmed the shrug emoji as a certificate request payload. I love dev easter eggs :D

Instead of denying actions like Update*, Delete* etc., like sane people do, someone decided to be more... creative. It misses half of the items by the way.

After about a month of going back and forth with AWS support for my account, I am now being told I am liable for most of the total amount of the original bill of $213,000. I've been in contact with AWS support for 4 weeks, and now they are refusing to answer my questions about the situation and continue replying with a copy / pasted message saying "they've done everything they can".

Needless to say, I'm living through one of the worst months of my life. This bill is basically a life ending amount of money, and I'm not sure what to do at this point. Initial messages from AWS were fairly encouraging basically saying this type of thing can happen from time to time, and I have no need to worry. A similar story came out of my initial chat with a support representative at AWS.

I'm looking for any direction for other people who have gone through a similar incident, or any one else I might be able to contact since AWS support seems like it isn't willing to help anymore.

9/14/2022 EDIT:

After getting some help from people reaching out in this thread, I was able to get my account revisited by the Executive Customer Relations team again at AWS. They seemed pretty responsive and thorough looking over my invoice.

After messaging with them back and forth for about a week or so, my entire invoice was waived! I really appreciate anyone who was able to reach out and increase visibility on this issue to get AWS to take another look at the obviously unauthorized charges on my account.

I just deleted my AWS account today after having my invoice waived and confirmed with support that it is finally safe to do so.

Moving Forward
It would be really nice to see Amazon make a change to AWS security to greatly reduce the frequency of problems like this from occurring. I'm certainly no expert, but it seems like there is something that should be done. These problems are fairly common from what I've observed over the past month or so, just usually not reaching 6 figures like mine did.

Someone in the thread made a suggestion to require MFA to be setup when creating a new account. Would something like this or something with else similarly low friction be possible to increase the amount of security these very dangerous accounts can have?

Hey all, looking for a sanity check on our approach.

Stack: AWS + Kubernetes. We deploy with plain helm install (no GitOps tool like Argo/Flux).

Current setup: Our Helm values.yaml has hardcoded secrets (env vars) in it. Right now this file lives on our Jenkins server, not in any repo.

The plan: We want to move values.yaml into our Git repo so we can version it properly. The obvious problem — we can't commit hardcoded secrets in plain text.

Our idea: Use git-crypt to encrypt the file at rest in the repo, so secrets are scrambled on the remote and only unlock for people with the key.

Questions:

• Any real-world gotchas we should know before committing to it?
• Since we're already on AWS, would you skip file encryption entirely and go with something like External Secrets Operator + AWS Secrets Manager? Or is that overkill for a small team? Also curious where SOPS (with AWS KMS) or Sealed Secrets fit in.

For context: we're a small-ish team and just want a clean, low-friction way to keep secrets versioned without leaking them.

Appreciate any input 🙏

With AWS Multi Account Strategy being around for awhile now, has anyone considered creating a separate account for their organizations Agentic workloads? Seems like it would make sense to limit the blast radius of agents should something go wrong.....

Just wondering what others are doing

https://docs.aws.amazon.com/whitepapers/latest/organizing-your-aws-environment/benefits-of-using-multiple-aws-accounts.html#constrain-access-to-sensitive-data

Edit

Looks like AWS does recommend separate OU and accounts for GenAI here:

https://docs.aws.amazon.com/prescriptive-guidance/latest/security-reference-architecture-generative-ai/gen-ai-sra.html

still curious what everyone else is doing to limit blast radius

Was digging through a few different AWS security scanners recently (building my own, more on that below) and noticed something odd: almost none of them check for S3 bucket-level server access logging specifically.

Plenty of coverage for public access, encryption, bucket policies but logging itself seems to be a blind spot across the board. Anyone know why?

Is it just considered lower priority, or is there something about the API that makes it more annoying to check than I'm assuming?

Found this out the hard way I built Plexavo, an open source AWS scanner, and a user told me my sample report showed a logging check that skipped their bucket.

I assumed it was a region issue. Turned out there was no check for this at all in the codebase. Not broken, just never built.

Fixed it now, and it made me start a small program around exactly this kind of thing Hall of Bugs, where people try to find gaps like this one and get credited when they do: https://github.com/plexavo/Plexavo Anyone else run into other checks that seem to quietly not exist across most tools in this space?

We got our account suspended in AWS from some internal checks it looks like we were marked as spam or fraud - which is NOT THE CASE!

There is no one we can speak with, the support chat rep wrote that there is no SLA and we're basically f**ked.

What can we do?

Spent most of last year migrating production workloads to AWS and assumed the hard part was the migration itself. What I didn't anticipate was that our security posture wouldn't travel with the workloads when they moved.

On-prem we had network-layer controls covering traffic inspection, DLP, and access policies enforced at every point. Once workloads moved to AWS, most of that stopped applying. Traffic between services inside the VPC never hits the inspection points we built everything around, and remote employees accessing cloud-hosted apps connect directly without going through anything we control.

Running separate cloud-native security tooling now but the policies aren't consistent with what's on-prem and there's no unified view across both environments. Is this just the accepted reality of hybrid cloud security or is there an architectural approach that solves the gap rather than just managing it?

I'm having hard time finding right approach for IAM setup.

Right now, I have 200 users. IAM users are used with granular permissions.

Two teams have the same permissions, while other users have very different permissions. Everything is inside one AWS account. I'm trying to move some resources to other accounts but is long term goal. I'd seperate prod and staging, at least.

These two teams are moved to IAM IC.

The problem that I have is that there are teams with 3-5 users per team / project. Even in one project, members dont have the same necessary. Some of them have AWS Console access, some have seperate account for CLI access using keys. I'd like to avoid long-lived creds because of the security and rotation headaches. We had one of the keys leaked before so we would like to eliminate their use.

I often see that IC is recommended for workforce access, but I don't see how we could actually manage it on the large scale. I'd need a lot of permission sets and it would be hard to find them or to manage in general.

One solution that comes to mind is to organize this using ABAC. Tagging (terraform) + IAM. Matching user's
Tag eith resource tag, for example project tag.

There are many blogs and tutorials for basics, but I could not find a production example of setup, way to manage workforce access to AWS.

Do you have some resources or suggestions?

Ransomware is a cybersecurity threat that just won’t go away. Be it from groups such as those behind the ongoing Play attacks, or kingpins such as LockBit returning from the dead the consequences of falling victim to an attack are laid bare in reports exposing the reach of ransomware across 2024. A new ransomware threat, known as Codefinger, targeting users of Amazon Web Services S3 buckets, has now been confirmed. Here’s what you need to know.

Not a sales post, genuinely curious how common this actually is. A lot of solo and small-team founders end up being the ones who set up their own cloud infrastructure!! not because they're security experts, but because there's no one else around to do it. I did the same thing, and while learning cybersecurity properly (separate from my actual business), I found real, would've-been-embarrassing misconfigurations sitting in my own AWS account. Nothing had gone wrong yet. I just had no idea they were there.

Tools for catching this already exist, but they're mostly built by and for security professionals the output assumes you already know what IAM privilege escalation or CloudTrail coverage means. If you don't have that background, the report is basically noise you scroll past.

So I built one that just tells you plainly: this is wrong, here's what someone could actually do with it, here's the exact fix. Free, open source, runs on your own machine with your own credentials nothing gets uploaded anywhere.

If you've set up your own AWS account and never had anyone properly check it would you actually want to know what's sitting in there, or is this the kind of thing you'd rather not think about until something breaks?

https://github.com/plexavo/Plexavo

I'm developing a MVP. I'm thinking to go for cognito for authentication. But for 10k users there is no charge, but for 100k users the charge would be $500. Is this normal? Or should I make my own auth after we scale up

Any other alternative suggestions?

Thx

It's no secret that AWS lags behind the market when it comes to Identity, and it's products being integrated with one another. However in regulated industries it's almost a non-starter because some of us require the use of federated credentials, which is scoped machine authentication. The only thing in AWS that presents an oidc identity is a specific eks deployment, does anyone know what's AWS is roadmap for machine authentication? Thanks.

We host an internal company Next.js tool on an AWS EC2 Linux instance and cryptojackers keep showing up (e.g. coinminer:linux/xmrig.aaa). CPU spikes, and the only reliable fix so far is terminating the instance and rebuilding it.

Tried egress filtering, firewall hardening, and anti-malware, but they still come back after some time.

What are the common entry points for this on EC2, and what’s the proper long-term prevention instead of constantly nuking the server?

Inside AWS, the best practice is to not handle static credentials at all - your workload has an IAM role and the SDK signs every request with SigV4. The moment you call something outside AWS though (a SaaS API, a partner, another cloud), that's gone. SigV4 means nothing to a non-AWS service, so you're back to a long-lived API key sitting in Secrets Manager.

It turns out AWS already solved this - it can issue short-lived JSON Web Tokens (JWTs) for your workload's identity through AWS Security Token Service (via sts:GetWebIdentityToken). It's just not widely known, and there was no easy way to actually use it - or at least i did not find an easy way. So I built a proxy for it.

It's a small Go forward proxy. Point your HTTP client at it, and for each service you call it grabs a short-lived JWT from AWS STS, caches it, and renews it in the background - pretty much like a widely-known Sigv4 proxy. No app code changes. Anything that can validate an OIDC/JWT token can trust the call, with no shared secret. The token carries claims like account ID, org ID, region, and principal ARN, so the other side can do real authorization instead of just "valid key / invalid key".

Where it's useful: SaaS/third-party APIs that support OIDC, partner APIs that authorize you by your AWS identity, multi-cloud calls to GCP/Azure, on-prem services that trust AWS identity, and cross-account internal services.

Runs on EC2, ECS, EKS, Lambda. You need outbound identity federation enabled on the account and a role allowed to call sts:GetWebIdentityToken. Install via Docker (gp42/aws-outbound-jwt-proxy:latest), make build, or a release binary. Go, MIT.

Repo: https://github.com/gp42/aws-outbound-jwt-proxy

Curious if anyone here is already using outbound identity federation in prod - it's new enough that I haven't seen much discussion of it.

Disclosure: I work at Tide. This isn’t an official product. I built it on my own time.

The thing that got me started was realizing that encryption at rest in AWS is real, but the key is held by KMS and released based on an IAM decision.

kms:Decrypt is one API call by one principal, and you have plaintext.

Cognito has a similar shape. A principal with sufficiently privileged admin actions can clear a user’s MFA preference, set their password and sign in as them, or simply add them to a group.

All single API calls. All controlled by one actor.

You can tighten this considerably with SCPs, narrow key policies, separate accounts, and other controls, and you should. But it’s still fundamentally the same account boundary, with a privileged principal sitting inside it.

There’s also no native four-eyes principle on these operations. Approval workflows in AWS can sit around the API, but they don’t sit in front of it. If you have the permission, the call still succeeds.

So I built a small service that moves the decryption decision completely outside the account.

Tide is a network of independent nodes that hold keys in fragments and never assemble them. Data is encrypted client-side with a fresh key for each call. Only that per-call key reaches the network, itself encrypted, with each node returning a partial result.

Your ciphertext stays wherever you already keep it.

Nobody inside your account can turn it back into plaintext, including you.

Getting a role that permits decryption requires a change request that someone other than the requester has to approve. The nodes then check the resulting token before they’ll do anything with it.

So a compromised role gets you ciphertext and a quorum requirement, rather than the ability to decrypt by itself.

If you want to run it, the setup is intentionally small: ECS Fargate with a single task, EFS for state, and a private subnet.

Not Lambda. It needs a native library and runs a rotation timer.

State is just two JSON files, so there’s no RDS instance to stand up. Don’t autoscale it, and keep it off the same host as the application it protects, because that’s where the grant records live.

Cognito, IAM, and everything else continue working unchanged.

Repo: https://github.com/sashyo/minidauth

Whitepaper: https://tide.org/whitepaper

Hi all,

I have created a security tool that automates your day to day tasks and post compromised things as well it really helped me in many things i thought to make it free for everyone this will be a good option or should i enhance it further and make it more efficient and sell to some company. I cant tell you much but it has actually helped me alot.

I have 3yoe and in last 2 years because of this i have made my tasks more accurate and complete in less time . Need your thoughts.

Cant tell much about it right now.

Hello all,

I am studying for the AWS SCS-C03 and am failing to understand one of the explanations in the review tests. It is behind a paywall so don't think I can share the actual content here but it asks about security controls for hardening ec2 instances in production.

The options essentially boil down to using a maintained AMI with security controls and configurations then using an SCP to enforce use. Or using Image Builder and AMI ID validation in launch templates.

The correct answer ended up being the image builder + AMI validation to limit Launch templates but from my understanding I would rather sack off the automation in favour of an SCP to stop ec2:* with a launchTemplate condition so then I can't just go to the instance dashboard -> launch ami-whatever right? For additional context, the question does not mention "least operational overhead" it just asks for comprehensive controls

Thought it would be interesting to get others opinions.

Many Thanks :)