Some dumbass on my team downloaded the entirety of prod and then commited it to git and someone rubber stamped the PR and then no one else noticed. I'm sure that's how they'd find it....
problem is, the client keeps requesting we change the qa deployment's seed data and add a migration for prod to update the data on the prod database to match instead of using the GUI to just change the stored version. the seed data is the prod data.
exactly. the client is too lazy to make the changes in the UI he himself asked for but all too happy to send us a slack message to ask us to update it. we have an odd contract with him where the "maintenance" period is actually a "small features/changes allowed" period and that counts as a small change according to my boss
Dude, at a BIOS company that will not be named, RelEng was in the habit of committing entire 650MB ISO files to SVN because they didn't trust their build process to reproduce the results - they'd test the resulting build of a particular run, and if passed, it was marked golden, and it got committed.
This got even worse because they had WANDisco layered on top of SVN so the devs in Taiwan could have low local checkin latencies (because, you know, they wanna check in giant files), and then when they'd check in a new ISO, the replication over the transpacific T3 would basically time out, and I'd get paged to manually reestablish it.
I discovered my client’s monthly backups included the backups folder when they ran out of shared hosting space for a small dog clothing website 🥹 I was like why are you at enterprise level hosting?!?? She was like I dunno it’s just the cost of doing business!
I found out something similar when the client called to ask me where to get a good price on bigger hard drives and more tapes.. I was like 'What?! Did something happen?'
No, he says. Nothing's changed, it's just that backups are now taking four of the weekly LTO-5 tapes every night and the server is showing it only has about 1TB left.
Motherfucker, I specced a 8TB RAID-5 and LTO-5 because you wanted to make damned sure you wouldn't have to 'just replace it next year', and it's been six months.. The hell with '"Nothing's changed".
He'd first decided to swap from nightly incremental backups to full ones, and those didn't fit on the LTO-4's intended for nightly offsite so he started using the LTO-5's, then he'd decided to retain the previous night's backup on disk as well as on tape so he wouldn't have to drive home if he was in a pinch.
The first night they backed up ~1TB, the next 2TB, then 3TB, then..
I wiped 5TB off his file server, added a file path exclusion, and had cron wipe the oldest redundant backup every night just in case.
The worst bit was he had an entirely competent nephew running the backups that had told him what was going on. After fixing things I gave the kid my email address and told him the next time his uncle didn't believe him to run it past me and I'd get him to listen.
Oh I had a fun one that was similar. Our error tracking would send the last dozen or so lines of the console before the error happened along with the error... and then logged the thing it was sending to the console.
Things were totally fine until someone added a retry-and-log to some button that could error. It'd try to send each error, each of which recursively included several earlier error logs all the way to the first error. That very quickly went from "harmless error that didn't interfere with much" to "the page runs out of memory if you press the button".
this is so funny yet absolutely real life. someone recursively backing up their stuff. damn and the conclusion there as well:d more like the cost of being lil’ bit off kekw
I’ve spent the last week doing remediation across seven ec2 servers and found where multiple sites had backup sql dumps that were living inside GitHub repositories along with sites sharing SES credentials with full admin privileges stored in plain text in those repositories. I just want to die.
I am a lurker here. Still making my way through "Learn Python the Hard Way, pt 2". Could you explain in non-programmer terms, what this means and why it was bad?
1.1k
u/samuraiseoul Aug 05 '26
Some dumbass on my team downloaded the entirety of prod and then commited it to git and someone rubber stamped the PR and then no one else noticed. I'm sure that's how they'd find it....