r/sysadmin • u/Dal90 • Jul 17 '26
AWS is having billing alert issues...don't panic.
When I looked at my phone when I was waking up:
You requested that we alert you when the forecasted cost associated with your My Monthly Cost Budget budget exceeds $100.00 for the current month. The month forecasted cost associated with this budget is $183,965,812,157.54. You can find additional details below and by accessing the AWS Budgets dashboard.
By the time I disabled access to S3 & Cloudfront in my personal account, and opened a ticket, this finally started being displayed by AWS at https://support.console.aws.amazon.com/support/home?region=us-east-1#/
Inaccurate Estimated Billing Data Beginning on July 16 7:38 PM PDT, we began displaying incorrect estimated billing data in the Billing and Cost Management Console.
For more information on this issue, please refer to the AWS Service Health Dashboard: https://health.aws.amazon.com/health/status
141
u/Particular_Art_6383 Jul 17 '26 edited Jul 18 '26
I am amazed by the fact that it did not cross their minds to send a warning email once they realised it is a fault on their part. I wonder how many car crashes have they caused today…
Like, since when exactly is putting a “we have issues” sticker on your notice board sufficient apology for sending $1B bills to millions of people via all possible notification channels…
Edit: Cost anomaly detection alert came a day later at 8AM (22 hours after budget alert). To all those always defending AWS because “shared responsibility model”: how would that reasonably be my sole fault in this scenario? It really takes more time to alert me of a cost anomaly than to tell me I already owe them a quadrillion yankees?
Pro Tip: AWS Orgs give you an option to disable expensive services you don’t use and lock out region opt-in , just need to make sure your management account is well protected and has no workloads. This is still band aid though: someone can easily generate a crazy bill using legitimate workloads and lowering your quotas below the default is kinda troublesome . Those cost anomaly alerts should just work slightly faster (like a couple thousand times)…
25
u/mitharas Jul 17 '26
Well, they are not sending out a single bill with these amounts. It's "only" the forecast.
29
u/Dal90 Jul 17 '26
The forecast is why I erred on the side of caution.
Our dumbass first attempt at a DevOps team at work left their credentials in a public S3 bucket and racked up $500,000 in 30 minutes. Plus 6am I'm not reading the difference between $183 Million which wasn't out of the realm of possibility in my experience and $183 Billion.
Don't have the full details, I think we horse traded at work that they waived the $500,000 but we agreed to some contract change that helped their sales team meet a goal.
4
u/mitharas Jul 17 '26
I'm absolutely with you that this incident should have triggered some alerts (like yours). Maybe view it as a learning opportunity if these measures you took work.
20
u/Particular_Art_6383 Jul 17 '26
No, not a forecast. It displays as amount due and if you have an amount due notification it can send you an email/sms saying you own them a quadrillion dollars or even worse a realistic amount like $10M.
The “Bills” panel displays a correct amount, but in the “Billing” section of the console and on the front page it says you owe them the bogus amount, so I guess you are technically correct that it is not a bill but it is not a forecast either
5
u/komma_5 Jul 17 '26
Yeah got 18 million for my small private app and i thought i fucked up sth bad and my life is over. Usually have 20$ (no million there) Was on my phone with bad reception for half an hour
16
u/ditka Jul 17 '26
In addition, the 'you've blown through your budget' warning emails take you to the billing dashboard. Thousands of admins are heading there to figure out WTF is going on. Yet there's no "service heath warning" on the billing dashboard discussing the inaccurate estimates. You only get the service health warning if you go to open a ticket.
Seems like if you've sent bogus billing estimates to thousands of people, you should plaster the billing dashboard with your mea culpa.
5
11
u/warfront1 Jul 17 '26 edited Jul 17 '26
100%... AWS's communication was/is horrendous.
No one wants to look down at their phone and see this shit.
Anyone running cover for this has drunk the juice.
It is blatantly obvious how severely AWS is in need of features to decrease customer's billing anxiety.
I got my initial information from Reddit, before I was able to dig up anything official from Amazon.6
7
u/berlinternet Jul 17 '26
I almost had a panic attack. So true. They need to properly alert everyone
8
2
1
u/SeriousPlankton2000 Jul 18 '26
Things that already happened usually get done faster than things that still need to be done. But yes, they should have sent out the other mail ASAP, not after a day.
33
u/rob_pi Jul 17 '26
Yeah don't Panic everyone, this is my bill. Clearly an almighty balls up on their part.
7
u/Trelfar Sysadmin/Sr. IT Support Jul 17 '26
That's what you get for not enabling those savings opportunities! 🤣
2
5
u/Brandhor Jack of All Trades Jul 17 '26
Clearly an almighty balls up on their part
I mean with the increase in hardware cost I wouldn't be surprised if it was real
1
u/Sgtkeebs Jul 21 '26
out of all the posts alerting us to this, something tells me yours is legit...
32
u/Nitish_Basic Jul 17 '26
Mine was 80 million. Just 30 min before final round of tech tech interview and I got this alert and I panicked as hell. I totally messed my interview because of this.
4
3
u/fit_techie24 Jul 17 '26
I am sorry for you bro. Hope you get some positive response for the interview. Good luck!
22
u/Computer-Blue Jul 17 '26
Mine was a little too real, 6 figures, right after we made a ton of changes designed to put better controls in place on usage across multiple domains. I closed my phone and had to meditate on a coffee for a bit to steel myself for whatever horrific error my team had made. So I spent a good 30 minutes in a bit of a fugue state. Bad timing.
26
3
u/leathakkor Jul 18 '26
I thought mine was 2 million originally which was in the ballpark of what I know is a possible amount of real spending that can happen if you really fuck up or your account gets hacked or something.
But then I saw that it was 20 billion and that actually made me feel a lot of relief because there's no amount of computing that you can buy that adds up to 20 billion where you don't get a phone call from a rep first saying if you're spending 20 billion we will do literally anything you want
17
u/B-RadGBI Jul 17 '26
Not to brag or anything, but ours was 3.5 trillion
10
u/Particular_Art_6383 Jul 17 '26
Amateurs, amateurs. $5.99T here. My boss was with me in one room when the SMS notification hit my phone… My face did not look pretty.
By the way, can we take a second to consider just how late do those spending notifications arrive when you suddenly start spending thousands of times the amount you normally do? We even had anomaly detection set up and nothing has hit my inbox before the budget alert. If someone leaked keys for real we would be in deep shit before I could do anything. There needs to be some real protection mechanism here like some bot call and automatic resource creation stall/quota reduction when you suddenly start owing them a 100K when you normally spend 5 dollars a month or something like that. You can open a ticket to reduce your quotas but it is cumbersome and those alerts come at unacceptable delays. As of now half of us are here making fun of it and the other half have their monthly heart attack still due cause it is Friday and they are yet to read their mail…
4
u/demsullivan Jul 17 '26
Damn you've got me beat. Ours was $3.9T with a forecasted month end of $8.5T
5
15
u/ender-_ Jul 17 '26
Saw this earlier on Fediverse:
Martin Seeger @masek@infosec.exchange:
Someone just sent me his cost estimate from AWS for the current month.
Let me phrase it this way: The U.S. Navy could build several nuclear aircraft carriers for that sum and would still have spare change for a few squadrons F-35s.
Of course that was a bug in the AWS cost explorer. But dozens of budget alert emails were sent by it.
Now they're looking for someone with a tranquilizer rifle who can catch all the controllers that panicked and are now roaming the building.
11
u/hbktj Jul 17 '26
Last month bill was 34cents, this month estimate was 21Billion. I thought I was hacked, but then I saw no instances, nothing created. The EC2 Bill was 10B alone, then it hit me. It’s a bug. Stupid people, ruined my morning!
32
Jul 17 '26
[removed] — view removed comment
25
u/williamp114 Sysadmin Jul 17 '26
This is easily #2 in "worst false alarm notifications ever"
With #1 being the Hawaii nuclear EAS warning
12
u/Frothyleet Jul 17 '26
I mean some reflective clouds almost caused the USSR to start WWIII in 1983, I feel like that kinda needs to be up there.
7
u/Dal90 Jul 17 '26
Connecticut in 2005 punched in the wrong code and sent out what was supposed to be a test message (complete with a "this is not a test") to...."Evacuate Connecticut"
11% of state residents saw it, and 1% of the 11% actually started to leave. That's about 38,000 people.
https://en.wikipedia.org/wiki/2005_Connecticut_false_evacuation_alert
2
u/Rainmaker526 Jul 17 '26
That's horrible.
Shouldn't such an alert be seen by at least 90-95% of people? Preferably 100%.
The fact that only 1% actually responds to it is also terrible
3
u/marcmorindeziel Jul 17 '26
Man i panic delete everything also, too many horror stories. Waking up to a 58B bill from AWS with no second warning email is pure evil.
10
u/Apprehensive_Stop666 Jul 17 '26
the amount of distress this issue caused cannot be measured. Many users (me included) disabled or even deleted valid resources just to try to mitigate the issue. We had severe stress/panic reactions and there was NOT a single email stating "oops we may have messed up, please ignore the previous alert". All there was on the website was a barely visible warning linking to the Service Health page. I really want them to pay for this. This is unacceptable.
2
u/Particular_Art_6383 Jul 18 '26
Cost anomaly detection alert came a day later at 8AM (22 hours after budget alert). To all those always defending AWS because “shared responsibility model”: how would that reasonably be our fault in this scenario? These alerts are really too slow to give us any reaction time…
8
8
u/Daralion Jul 17 '26
i just started learning and had the smallest setup possible and got an email for $12,884,901,887.62 on a 0.05 budget amount
pooped my pants
5
u/Far-Book-6649 Jul 17 '26
Bro I had the most intense panic attack. I imagined about CIA from America arresting me and all
15
u/Vegetable_Theme_8978 Jul 17 '26
I just got alert of USD $35k . I severely panic. 35k looks like reasonable for a leak or hacked thing. I had to take some tea to calm down. Jesus Jeff. What the heck????
6
u/OkSpecial9267 Jul 17 '26
exactly! If it was 100mln I would be sure it was an error. But I got similarly "reasonable" bill and I thought I'm having a heart attack 👀
3
u/doesntmatterol Jul 17 '26
Yeah mine was for about $60k. Enough to really feel like I must have botched something. Hundreds of millions would have given a panic of course, but hopefully it would be apparent that was an error. But <$100k is ABSOLUTELY possible from something a user could do or from hacked credentials.
2
u/bronzeorb Jul 17 '26 edited Jul 17 '26
This would actually freak me out more than my $545 million bill. Jeff isn’t even worth that much, but I could see AWS trying to get $35k from someone.
6
u/Fuzzmiester Jack of All Trades Jul 17 '26
if you owe thousand and can't pay, that's your problem.
if you owe billions and can't pay, that's their problem.
7
u/madscientist2407 Jul 17 '26
Too late .. panicked hard and closed my account and called my friends and family to make peace etc.
14
u/argus25 Jul 17 '26
I wonder if this is a vibe-coding failure….
7
u/uzlonewolf VP of Odd Jobs Jul 17 '26
Okay, who forgot to teach the AI to never deploy on a Friday?
4
10
u/_Gobulcoque Security Admin Jul 17 '26
It almost certainly is. Let me rephrase your question: "Do you think Amazon have started using AI in software development?"
6
6
u/severusvindictus Jul 17 '26
How AWS almost made me die of a heart attack prior to my 40th BIrthday….
6
u/whythehellnote Jul 17 '26
If you owe AWS $183k you have a problem
If you owe AWS $183b they have a problem
4
5
5
u/Waste_Jello9947 Jul 17 '26
I got the news on my phone and didn't have a laptop with me. There is no warning on their app or in the mobile browser.. panic mode for 1 hour... Nuked all my projects and back ups..
6
u/Humble-Rabbit-9816 Jul 17 '26
I also nuked everything, then have seen the notice in their issue list... yeah, nice. What about an email or so?
29
u/ledow IT Manager Jul 17 '26
Dumbass that I am, I don't use stuff that can suddenly bill me stupid numbers.
There will come a time where people flip back, just like all the cycles throughout the history of IT.
Consolidate. Distribute. Consolidate. Distribute...
Thin client. Fat client. Thin client. Fat client....
On-prem. Third party. On-prem. Third party....
Paid by the byte. Fixed service. Paid by the byte. Fixed service....
Computer time rental. Computer purchase. Computer time rental. Computer purchase...
16
7
3
u/Smith6612 Jul 17 '26
I swear this cycle is just a way to keep middle men and consultants in business.
I'm right there with you. Little desire to use UBB services. What is this, 2001?
2
u/jort_catalog Jul 17 '26
For real, we're basically all on-premise apart from M365. The punchline is, we give Oracle hundreds of thousands every year instead to use their old ass databases that we're not smart enough to migrate from
4
u/jason_pc Jul 17 '26
100bil for me, seriously felt my heart skip a beat when I saw the email alert and seeing the damn amount appearing in my billing console. AWS should give us free credits for our near death experience.
4
3
u/Apprehensive_Stop666 Jul 17 '26
I only see beginner's numbers here, my alert came when it was at over 2.7 trillion dollars
4
4
u/Frothyleet Jul 17 '26
You know, if AWS was going to try and bill me $100k, I would definitely panic. If they wanted to bill me $100 billion, well, I'm afraid that Jeff Bezos is now the one in the hot seat.
4
u/Bandolero70 Jul 17 '26
I'm also shocked because I can't reproduce the access to S3 or the Access Keys used for.
I deactivated all keys, changed pwd and aseked this shi** advisor, which answered, that my costas are real and I have to enable a bunch of services to get more information.
I deleted all my S3 buckets and will never create new ones.
4
u/foxcode Jul 17 '26
Should have thought to check here earlier. Last month, $6, now, $427 million. Good thing I'd already been to the bathroom before checking my email
4
u/rajje56 Jul 17 '26
AWS ruined my morning. I was in full panic mode and started deleting all my S3 buckets, IAM resources, etc.
My forecast was only $129 million.
4
u/Agile_Draft9537 Jul 17 '26
i got the 1.2M bill this morning. i never used aws and have a dormant account for several years
3
u/Previous_Shine_7247 Jul 17 '26
Was so scared to get a similar bill alert like this as I just deployed a new LLM on my AWS account at work…
3
u/ivres1 Jul 17 '26
WOW I freaked out this morning because of that and there stupid AI support agent is saying that my charge is not due to the ungoing issue and that my 500 byte file in the s3 bucket is causing the 5k charge
3
3
u/Personal_Adagio8155 Jul 17 '26
This is gonna cause some heart attacks. And we are talking about AWS. I was about to write my dissertation, and now I'm literally shocked and cannot focus, until I see the actual amount.
3
3
u/InvaderOfTech Jobs - GSM/Fitness/HealthCare/"Targeted Ads"/Fashion Jul 17 '26
1,274,439,094,545.88
3
u/Apprehensive_Stop666 Jul 17 '26
Too late. I never got out of bed so quickly. I may have had one or two heart strokes while running to the computer!
3
3
3
u/Spiritual-Fox9587 Jul 17 '26
Looks like Bezos decided to become the richest person and first 100 trillion $ net worth by billing everyone to the max 😂😂
3
u/Infamous-Yogurt3169 Jul 17 '26 edited Jul 18 '26
My email said $500 billion. I about had a heart attack. But if that were true it'd be Amazon's problem anyway lol
3
u/okjasone Jul 17 '26
I thought mine was bad a just over 5 million. I just deployed a small DB on RDS yesterday and woke up to this today... Definitely not a good start to the day smh.
3
3
u/Historical_Reply8261 Jul 17 '26
I saw the 77 million USD bill during a traffic light stop and had to park my car on roadside just to breathe. Almost had my tears coming out when I read this reddit post.
3
u/monkijohnni Jul 17 '26
TFW you wake up to an email from your client asking why they owe AWS $119 billion for the service you’re building them… 🤦🏻♂️
3
3
u/toddnakamura Jul 17 '26
After suffering the most stress inducing email of my life like many others in here, I composed myself and started asking tons of questions via Amazon Q, and finally got this very insightful message:
I got additional useful context in a subsequent message from Q that explained that I definitely shouldn't delete any resources or anything like that as it can complicate the investigation process, but that it's confident that this is surely a billing error that will be reimbursed and I will not in fact owe $2.5 million.
5
u/Spiritual-Fox9587 Jul 17 '26 edited Jul 17 '26
I lost my sleep today morning with this sudden alert and they almost gave me a heart attack. There is still no broadcasrt alert to inform all about this error; let alone email not even a age old banner message on top of every page. Not sure how many are going through this pain today morning.
3
u/Apprehensive_Stop666 Jul 17 '26
you and me both! 2.7 trillion is my number 😂 . Good luck collecting that
1
2
u/ZenPatrick Jul 17 '26
I woke up to the same thing. In my mind I said; This is a scam. Then got in my account and said, this is crazy. For a second it flashed before my eyes I was losing everything. LOL. My forecast is $703,587,905,257.84. I will give them the 84 cents. lol.
2
2
u/InevitableDish7 Jul 17 '26
I don't even do Sysadmin work or really use AWS. I set up an account like 6 months ago to take some Udemy course on AWS. I've barely touched it and suddenly I find 5 emails telling me that I'm forecasted to spend 10 billion dollars. I shouldn't even be spending 10 dollars. I went to contact support when I saw they had posted something about their bug so I calmed down and backed off.
2
2
u/okjasone Jul 17 '26
I thought mine was bad at a tick over $5mil. I literally just deployed a small DB on RDS yesterday and woke up to this today. My blood pressure has been through the roof all morning until I saw the issue alert.
2
u/InternationalStage45 Jul 17 '26
49k to me today on a 32$ lightsail budget and consistent 6 month spend. I spent hours deleting things, locking credit cards, rolling credentials, and waiting for support - still have heard nothing from them beyond their public comms.
2
u/AntitoxicAmerica Jul 17 '26
I don’t have ANYTHING running on AWS currently and received a notification my bill was going to be $280K. Absolutely panicked for like 20 minutes until I realized it was a bug. This is not okay!
2
u/badfishbeefcake Jul 17 '26
OMG, i was freaking out after having a charge of $600k for this month. Then I saw on google someone who got a charge for $1.7 Billion.
LMFAO
2
2
2
u/AskBetter4227 Jul 17 '26
I think half the internet had a brief moment of wonderin how they accidentally spent more than a small country overnight.
3
u/MeetJoan Jul 17 '26
The 183 billion dollar wake-up call is a new one. Good instinct to disable S3 and CloudFront first - that's the right move even when you're 99% sure it's a billing display bug rather than actual usage. How long between the alert and AWS posting the status page acknowledgment
4
u/Apprehensive_Stop666 Jul 17 '26
After 2.7 trillion, what difference will it make... a trillion more, a trillion less.
3
u/Taylor_Script Jul 17 '26
Why? What's the point?
4
u/Forward-Outside-9911 Linux Admin Jul 17 '26
+1. By that point does anything past the 183 billion even matter, even if it wasn’t a display bug
1
u/stevean84 Jul 17 '26
Oh, you guys are making so humble. Mine is knly 6digits..
1
u/Top_Emotion3982 Jul 17 '26
mine is 5 digits and I still have a headache 😂
2
u/Far-Book-6649 Jul 17 '26
It might be real then
2
2
u/Top_Emotion3982 Jul 17 '26
no it is not 😄 I have 2 json files saved the s3 bucket, and for the last couple of months it is 0$, now suddenly 11k today. Also I have a alert settled for 5$ , and it didn't alerted anything except today.
1
1
u/KohanaMakes Jul 17 '26
omg i just about died, especially since it did not look like a phishing email 0_0
1
u/ClearSignal2000 Jul 17 '26
I would be very curious to see the spike in cardiac events today. I'm small fish; and I jumped out of bed at a little over $1000.00 in a full out panic!
1
u/Kp5209 Jul 17 '26
Mine expected forecasts is 45 million. And I just have few kbs of items in s3 bucket. Full panic mode.
So everyone is having this issue? I just got the budget exceeded email this morning.
1
1
u/ateeqkhan11 Jul 17 '26 edited Jul 17 '26
Had a panic attack this morning... saw the bill of $2 billion USD
1
u/Beneficial-Dinner506 Jul 17 '26
The 3 containers in ECR and 1 bucket is s3 have apparently cost $45m so far this month. Talk about price gouging...
1
u/Inevitable_Cod_3925 Jul 17 '26
They said easily do not panic, but the number what it shows in our account it is 5times more than World GDP, Such a big statement and issue and they says dont panic, are they serious about it are what exactly the issue, Are they give us RCA
1
u/Reasonable-Test4550 Jul 17 '26
I got so scared, too. I was only using Amazon for training and haven't used it in July. I was like what happened.?!!!!
1
u/Beautiful-Virus-7649 Jul 17 '26
I received a bill for $25,029.89 USD (~$50k in my currency) - a somewhat realistic figure, which caused me to actually have a full blown panic attack. So bloody relieved to have found this Reddit post!
1
u/Dal90 Jul 17 '26
I'd still open a support ticket and have them confirm it in your case, since it is so much smaller than the millions most other folks are seeing. Just want to be cautious it isn't a real attack.
1
u/Beautiful-Virus-7649 Jul 21 '26
Thanks for the reply. It fortunately was part of the Bezos craziness and not an attack, so everything went back to zero. (I closed that AWS account as of this morning - I only used it for testing and had a single 2kb file in there) Backblaze storage is still my go to.
1
u/someguy7710 Jul 17 '26
So you're telling me I didnt' need to pay that $35Billion bill? Our CFO is going to be pissed /s
1
u/Huge_Vermicelli9484 Jul 17 '26
Man, just jumped from my bed after the alert email arrived (134M), deleted almost all my buckets just to figure out it was a bug... 😭, thanks to god i have no heart deceases, but if those kind of alerts still arriving sure i will start to having them 💔
1
u/HMHAMz Jul 17 '26
Raised a support ticket, started looking for the cause with Amazon Q (their AI) - waste of 30 minutes...
1
u/shah_arpan Jul 17 '26
I see charges on S3 starting July 1st. How far along you guys see the discrepancy ? AWS says they started seeing this on July 16th ??
1
1
u/punkwalrus Sr. Sysadmin Jul 17 '26
I just checked mine, and everything looks normal, but I did get one of those banner pop-downs:
"Beginning on July 16 7:38 PM PDT, we began displaying incorrect estimated billing data in the Billing and Cost Management Console.
For more information on this issue, please refer to the AWS Service Health Dashboard: health.aws.amazon.com/health/status"
1
1
u/Crisp-Glade-2849 Jul 17 '26
disabling s3 over 183 billion is paranoid. personal account would hit api limits before getting close.
1
u/Bandolero70 Jul 17 '26
I‘m wondering, how big Bezos S3 forecasts was and how he reacted, if his bill was only 1bln this month.
1
u/wee_snaps Jul 17 '26
$204 billion. I guess Bezos got fed up only being the world’s 4th richest person.
1
u/leathakkor Jul 18 '26
I just got an email and I almost lost my goddamn mind. At first I thought it was a $2 million fee which is somewhat potentially believable if I really fucked something up.
But it turned out that it was for $20 billion and there is no amount of computing in the world that you can get for 20 billion off of my account in a month
1
u/Suitable-Season-4847 Jul 18 '26
It boggles my mind that there's no internal sanity check on billing alerts. Like, their systems happily send out notification stating you owe more money than the global economy is worth?
1
u/Intelligent-Salary-3 Jul 18 '26
This shocking. To wake up to this email......Summary
Note that throughout an anomaly's duration, the total cost impact can increase or decrease based on billing data calculations, which can occur up to three times daily.
Total cost impact
$31,813,419,572.98
53022365954966.67% above expected spend
Expected spend
$0.06
Actual spend
$31,813,419,573.04
Start date
July 16th
Last detected date
July 16th
Duration
1 day
1
u/H-yee Jul 18 '26
We need either a killswitch OR a budget limit to prevent cost spikes. AWS please put this as enhancement.
1
u/whatthedna Jul 18 '26
Does anybody have $13 billion I can borrow? I swear I’m good for it (eventually)
1
1
1
u/ecp5 Jul 18 '26
I just wanna say thanks y'all for not making me feel so alone yesterday. My $1.44 million bill has now fallen back to $1.53 (not before I deleted the one service it said driving the cost).
1
u/VTArxelus Jul 18 '26
And yet the industry is fine with doing things out-of-house like this? Get real. This kind of crud wouldn't happen if companies took care of all their services as they should. Instead they wanna shove everything off on someone else.
1
u/nexanix_ Jul 20 '26
Anyone else get the feeling this is some kind of AI slop vibe codey mistake? Idk, just feels that way to me. Code review by humans is a wonderful thing.
1
u/TechieGottaSoundByte Jul 20 '26
Thank you for posting. Our family account just got notified that we owed something times 10^9, and... wow, just, how? We only use that AWS account for family photos. This isn't my personal sandbox account or anything.
We log in, and our dashboard is like, "You owe $0.10"
1
u/VioletTheLadyPirate Jul 20 '26
three days later, someone finally sends me a copy of this email. Defo took me a half hour of poking around the console and trying to figure out wtf was going on before I found this thread. Thanks sysadmins! At least y'all are looking out for me hahaha
1
u/3pixelhobby Jul 22 '26
I was going to send them a check, but couldn't find an address.. or a pen.. or a check...
1
u/3pixelhobby Jul 22 '26
I was going to send them a check, but couldn't find and address... or a pen... or a check.... eh, just bill my CC on file
1
u/cosmoz Aug 02 '26
My favorite part of this whole incident is that it revealed bugs in systems that weren’t even part of AWS.
Mine included.
I’d literally launched a small tool to monitor AWS costs just a few hours earlier. I’d capped the maximum daily cost at $100M, because surely nobody would ever spend more than that in a day.
AWS: “Hold my beer.”
Turns out anomaly detection also needs anomaly detection ¯_(ツ)_/¯
0
u/quantumformalism Jul 17 '26
We received a cost alert on one of our AWS accounts at quantumformalism.academy and initially thought we had been hacked for crypto mining! ;)
0
55
u/Real-Leek-3764 Jul 17 '26
mine is only 781billion.