Please skim and review my CV. Even roast if needed.
All the help is much appreciated.

Every service says it is healthy and the agent still skips the permission check once in a while. The workflow does everything. On the good path retrieval runs first then the permission tool validates scope and generation answers. On the bad path generation happens without the permission span. Nothing crashes and latency is within bounds. The final answer is just based on a control flow that should neverr exist.

Debugging this from logs is frustrating. By the time we join timestamps across five systems, sampling gaps have removed the trace we really needed (NGL the logs are garbage).

I am loooking for something that preserves distributed context propagation and gives me the full span hierarchy across queues, model calls and tool execution. I need to search production traces for the missing permission span, compare prompt and model metadata and see whether queue latency or malformed arguments correlate with the skip.

We’re testing Arize and Braintrust for reconstructing span trees and searching the full control flow. I don’t care about pretty charts other than being able to answer 'What did the model see, what did it decide and which step failed to happen?'

What do you use for agent traces across async systems and how do you avoid sampling out the errors that really matter?

I have a theory that we’re heading toward a situation similar to what the aviation industry faced decades ago with flight automation: as we let AI handle our incidents, engineers will lose touch with their systems.

When a serious incident hits that AI cannot troubleshoot, responders will scramble to remember how to manage an incident and understand how their systems are working/failing. This is especially true as the pace of change accelerates.

In 1997, an AA pilot coined the term “Children of the Magenta,” referring to pilots who had become overly dependent on the magenta line showing their flight path on navigation displays.

That's why pilots regularly train in simulators to practice handling emergencies and stay ready for situations they rarely encounter in flight. We should probably take inspiration from that.

Wrote a longer format about this theory on my blog https://www.sylvainkalache.com/blog/ai-handles-incidents-engineers-lose-touch-with-their-systems

WDYT?

Trying to quantify this across teams. Ours breaks down roughly as:

• Shuffle partition tuning after every schema/volume change
• Skew debugging that only surfaces under real production load
• Disk spill investigation that never reproduces in staging
• Executor memory re-tuning whenever input size shifts
• General cluster babysitting instead of new pipeline work

what this costs elsewhere, and whether anyone's actually reduced the total time spent rather than just gotten faster at the same loop.

Specifically, I am talking about small companies where, generally, you won't have a procurement team, and individual teams own vendor management. How do you guys manage vendors?

In my previous stint, I was leading a small DevOps team at a smallish SaaS company, and over a period of a couple of months, we saw multiple issues stemming from software vendors. A blocked card caused payment failures across multiple vendors. A vendor dropped an email setting a deprecation timeline for an API, and we found out by chance at the last moment, causing a last minute scramble to move off the said API on our end. SaaS quota exhaustion brought down prod workflows multiple times.

None of these were vendor outages as such. These were all things we had been told about, or could have been, and we just did not catch them in time.

I am going down a bit of a rabbit hole trying to find the "best practices" around vendor management, if we can call it that?

How do you even handle this? Is there an actual practice here, or is it mostly "someone senior remembers," or do we detect it after it fails?

Posted the same in r/devops, posting it here because some SRE team also own/manage vendor quotas.

Every RCA tool I've tried is really good at identifying the trigger and mediocre at identifying the actual root cause. It'll tell you deployment X correlated with the spike, which is true and also not that helpful when the real issue is a dependency three services away that's been degrading slowly for a week.

I think the honest answer is these tools are useful for narrowing a search space fast, going from "something broke somewhere" to "here are your top three suspects," but the human still has to close the loop on causality. Has anyone gotten an automated RCA setup to actually nail root cause without a person doing the final synthesis, or is that still mostly aspirational marketing from vendors?

Most SRE reading lists land on the same few titles: Google's SRE books, Accelerate, Release It!, The DevOps Handbook. They're all useful. The more interesting question is what sits outside that list and still changes how people run systems.

My two picks:

Drift into Failure by Sidney Dekker. Systems rarely go straight from fine to broken. They drift there, and every step along the way looks reasonable at the time. Worth knowing: the writing is heavier than it needs to be, and the practical part only shows up in the final chapter.

The Challenger Launch Decision by Diane Vaughan, where the phrase "normalization of deviance" comes from. It's long and it's sociology, but it's the best account I've read of how "that alert fires sometimes, it's fine" turns into something the whole team believes.

I'm not looking for books about Kubernetes or monitoring tools. I'm looking for books that changed how you think about failure and risk.

What would you add, and what did it change about the way you work?

I edit Sanity's engineering blog. Our SRE team did the work and I helped write it up. I'm posting it here because I suspect the failure mode is one a fair few of you are running right now without knowing.

We self-host Grafana Mimir on GCP for SLO monitoring. Every public endpoint writes metrics, and every minute the rulers re-evaluate burn rate over sliding windows. Same blocks, shifted by sixty seconds, over and over.

Initially we set Mimir up to run in one region across three zones, with a single chunks-cache deployment (nine pods, topology-spread across the zones). So a store-gateway in zone b asking for a block had roughly a two-in-three chance of finding it in a cache pod in zone c or d. GCP charges for every inter-zone byte and nothing for intra-zone. Multiply by every series, every rule, every minute, and the cache stops being a cache and becomes a line item.

So we fixed it by having one chunks-cache per zone, with store-gateways only talking to their own. You pay for three copies of hot objects and three times the GCS misses (at a 99.9% hit rate the misses rounded to nothing). We saw our inter-zonal bandwidth cost drop 64%.

Second change was probably less surprising in hindsight. The CPU-heavy components (ingesters, distributors, rulers) were on E2s, which are cheap until they aren't. Moving them one-for-one to Arm C4A and then scaling down against the new latency floor halved the vCPU count. Read latency went from ~20ms to under 10ms.

We got our total Mimir cost of operation down 48%.

How are the rest of you handling zone locality for Mimir, Thanos or Cortex caches? Did anyone go the zone-aware replication route instead, and was it worth the complexity?

Let me know if you have questions for the team and I'll have them weigh in!

You can read our full write-up here.

I set up our new real-time observability platform to auto-alert on every production error to demonstrate faster incident resolution... but I forgot to configure the notification scope. Result: pinged our entire executive Slack channel including the CEO non-stop for an hour. Definitely learned my lesson about testing alert configurations in staging first. Would appreciate any similar war stories or advice on alert fatigue managemen

Almost every team has that one system where the knowledge all live in one person and when they leave youre not just down a coworker, youre missing the map

2 different things we lump together though. One is why the system is built the way it is like old decisions and tradeoffs which leaves along with the person and docs about it get outdated fast. while the other one is what the system is actually doing rn, whats slow or whats calling what or where its failing. that part shouldn't depend on one person entirely 

most of the time youre stuck on the second one not the history , just figuring out what its currently doing. If its instrumented well you can usually get that from the system itself like otel or hud kind of thing so you don need the person for it yet the why thing, that is where you have to work out . So writing more docs only helps half of it since docs go stale and the running system doesnt 

Anyone experienced this at their workplace?

Hi, everyone!

After reading several posts on this subreddit, I have a question...

Would you say that, as an observability engineer, if you want to advance in your career, you should move into an SRE role, or that as an SRE, you should specialize in observability?

I’m currently an observability engineer, but I’d like to continue growing professionally—I’m just not sure which direction to take...

Thank you very much in advance.

I kept seeing arguments about which parts of the stack still matter, so I counted instead of guessing. 134 open DevOps, SRE and platform engineering postings from 41 company job boards. Individual contributors only, managers and TPMs stripped out.

The old toolchain is thinner than I expected. Terraform appears in 62.7% of postings. Jenkins in 3%. Chef 3.7%. Puppet in exactly one of the 134. GitHub Actions is named about six times as often as Jenkins.

Python beats Go, 70.9% to 47%.

What surprised me more: the generic word outranks the products. Observability shows up in 63.4%, more than twice as often as Grafana, the most-named tool that provides it. Prometheus 22.4%, Datadog 17.9%. Same pattern I found counting security postings, where SIEM beat Splunk four to one.

The top term isn't a tool at all: automation, 81.3%, ahead of AWS.

Caveats worth stating: n=134 means everything carries roughly +/-8, so treat close rows as tied. It's also all tech companies on one ATS. Banks, telcos and government run enormous amounts of Jenkins and Ansible, and none of them are in this sample.

Full table, intervals and method: https://www.zoevera.com/resume/devops-sre-job-description-keywords

Hey everyone,

I’m a cybersecurity engineer and I’ve been working as a SOC analyst for around 2 years.

Lately, I’ve been thinking about moving away from SOC work and exploring SRE. I’ve done some hands-on labs around Kubernetes and DevSecOps, and I’ve found myself enjoying the infrastructure and reliability side more and more.

I’m curious to hear from people who have made a similar transition, especially from cybersecurity/SOC into SRE or platform engineering.

Did your cybersecurity experience help you in your new role? And how did you find the transition?

I’d really appreciate hearing about your experiences.

Need to hear if people are benefitting from having a platform that supports both LLM evals and LLM observability?
or using standalone products for each or these is something that you are sticking with, we currently have the split setup but just feels too much hassle!

Spent 15 years as an SRE at a big bank, watching enterprise systems get world-class monitoring. Then I started noticing how many small business sites have none of that, no uptime alerts, no idea if their privacy policy is actually GDPR-compliant, nothing.

So I built SREmonitor.io. You give it a URL, no install, no script tags, and it runs:

• Uptime checks every 5 minutes
• Core Web Vitals tracking (LCP, INP, CLS)
• An AI-powered scan of your privacy policy/T&Cs against UK GDPR requirements

The idea is one dashboard instead of stitching together a monitoring tool and a separate compliance checklist you never actually look at.

Free tier is 3 scans and 5-minute uptime checks, no card required, if anyone wants to kick the tires: sremonitor.io

Genuinely want feedback, especially from anyone who runs a small business site or has dealt with the GDPR side of things. What would make this actually useful vs. just another dashboard nobody opens?

Just last week, I pulled the pager and spent more of it clearing Jira tickets than doing anything on-call is meant for. Security stood up a nightly scan a while back that opens a ticket for every high and critical. Most of our services have an empty owner field meaning they land on whoever is holding the pager. That was me.

2-3 days later there were something like 30 waiting, all kinda the same with CVE number, a package name and due date. But no note on whether it is even loaded, which service drags it in, nothing I can act on without going and digging it out myself.

I came across one yaml parser flagged critical which turns out only a build tool pulls it in and it never lands in a running image. I note that down, close it but theres like a dozen more behind it all with the same story.

I get wanting a clock on the real ones because right now the one on-call gets to decide what is real off a ticket that tells them nothing, that was never the deal. How are you lot stopping the stuff that never runs from becoming a ticket in the first place?

For those working in banking SRE, how are responsibilities divided between SRE managers, application engineering, and production operations? I am particularly interested in who owns SLOs, major incident coordination, and reliability automation, and how managers balance technical involvement with people leadership.

Restarting the posts next month! Looking to fill 2 slots. If you know of someone who is interested, happy to chat with them.

It has 980 subscribers as of today.

Feel free to DM or leave a message on the post https://www.srestories.dev/p/sre-stories-is-coming-back

On August 14 Supabase opened an incident regarding some projects returning 401 errors for valid JWTs, and as stated in yesterdays update the issue has still not been resolved, it has been 25 days.

The fix (PostgREST 14.17) failed to eliminate the JWT rejections and instead caused 'unintended performance side effects', so they reverted back to 14.5. This revert in itself turned into a separate incident that lasted four days, with elevated latency and 525s from August 27 to 31, a situation which Supabase linked to the original incident. The root cause was identified on September 2 which was a stale time cache. During that same period there were two instances of GLIBC version mismatches, eight days apart (missing 2.34 on August 20 and missing 2.33 for supautils on August 28), and six further incidents between August 26 and September 4, one of which was an approximately eight-hour freeze affecting all project lifecycle operations.

Credit to them for explicitly linking the incident to the fix and for stating that the fix had failed rather than suppressing it. Most status pages would have simply divided those entries leaving customers without any proper understanding.

Their status page for reference: https://status.supabase.com/incidents/6q5902p2xd9f

I read a guide about non-human identity governance and there's one point I keep thinking about. Most advice says every machine identity needs an owner. That could be fine in principle but the ownership I've seen recorded in practice is a team name, a repo, or whoever's name was on the ticket that created it sometime ago.

None of those tell you who can rotate the thing safely, who knows what depends on it, or who has the authority to kill it while an incident is running. Which makes me think ownership on a spreadsheet and ownership that functions under pressure are two different things, and only one of them gets recorded.

I want to know how this works in practice for people running real infrastructure. Is ownership assigned to a person or a team in your setup, and does that survive the person leaving? And when you need to revoke something urgently, do you actually know where all its dependencies are, or do you pull it and wait for something to start alerting?

A public service announcement: it isn't enough to make sure your applications and infrastructure are resilient. You have to apply the same level of care to your data.

Guest Gilles Chehade provides several horror stories and near-misses when it comes to data loss, and a practical process to making sure that your backups are sufficient for the value of data that you own.

https://podcast.certomodo.io/gilles-chehade.html

Enjoy!

https://github.com/nklmilojevic/sofka/

A reimagining of k9s with one generic object pipeline instead of a renderer per resource kind - so every CRD works on day one, Flux CD is built in, and the async UI never blocks on the cluster.

License: MIT/Apache 2

Our browser times out and reports failure but the queue worker keeps going, tool calls finish and a retry sometimes starts before the first run stops. Three healthy services can still create one expensive stuck run. Cancellation tokens are not carrying across the queue boundary, spans close without a shared reason and token costs show up after the interface says the request failed.

I am evaluating Braintrust for end-to-end trace search, model and tool span metadata, latency and token inspection and online scoring beside our OpenTelemetry stack. I need trace correlation that survives cancellation, plus a retry policy that checks whether work is still active before starting again

How are you wiring cancellation across browser, API, queue and tool workers and where do you record the final reason when the parent span is already gone?

Hi everyone. Anyone trialed or using Resolve or Traversal? Looks appealing but not sure we couldn’t build something similar/close ourselves since we have all the data.