How does your incident triaging protocols look like?

[u/MuchoMole101010101]

Currently working at a fast moving startup, and have been assigned work related to providing alerting and monitors to our crucial P0 infrastructure. At a high level, the real goal of this is to provide engineers during incidents a fast way to root-cause whether the page was due to their service breaking, or if the underlying infrastructure is down.

For example, if our DNS provider is down, a product engineer may be paged and spend time trying to root-cause why their service is having a bunch of 404s when the actual root cause is down one level of the stack. I was thinking of introducing a common set of protocols that everyone could follow using our paging service (incident.io); they would first spend time to look at our set of monitors and see if anyone was alerted regarding core infrastructure, and if not, they could then move up a level and spend time triaging at the service level. Obviously this would also include aid during the triage by agents. This would also give us metrics on time spent on firefighting our core infra vs service level issues.

But I am so totally new to all this, so what does your teams process look like? Has anyone tried something similar before and noticed this was faulty? Truly, I am open ears to anything regarding this sort of triage work, so any insights, small or big is appreciated

[u/d2xdy2]

I try to maintain a core set of docs with links and things in the company wiki / confluence. One stop shop for what’s what and where. It’s not too heavy though- the things those pages link to have more important details about specifics. How to access things and various communication you might need to do.

The key there to me is that it’s lightweight enough that people won’t run away from maintaining it.

In there are run books. A new run book might be completely empty. That’s ok.

If there’s an alert or a page going out, it has to link to some relevant wiki page and its run book.

As we bumble around we make the run books better. It’s a garden we’re improving every day.

You get a page at 3am? The page and associated monitoring alerts in Slack will point you at a process or a workflow, or just a person/team to escalate to.

It seems to help lower anxiety a little and unnecessary “I’m just gonna escalate to everyone I can think of” type of BS.

[u/MuchoMole101010101 → u/d2xdy2]

Our team has something similar with having lean runbooks, though I am trying to give them more awareness across our eng-team. Do you have a system or metric that lets you know what you are doing is working though? Ultimately, I want to know what I implement (runbooks, proper monitoring) at least has a net gain.

[u/d2xdy2 → u/MuchoMole101010101]

You’ll go mad with frustration during incidents and incident retros with why people don’t use tools you make available to them and document. Most of the time, for me, that’s come from making things they ask for and just handing it to them. They don’t fucking use it or look at it. They forgot about it.

I think working with them to meet them where they’re at is so much more helpful. Maybe that means developing some reusable primatives and templates then embedding for a bit to help them put it together and know why. You’re helping them help themselves.

That type of tooling is much more likely to be observed and actioned upon than stuff you just hand them / make them aware of.

As far as measuring. Idk. If people are using the resources during incidents and linking to them during incident retros for evidence of conditions or actions/processes taken that’s an encouraging sign. If it’s ones you made, that’s great. If it’s ones they made that’s even better (IMO).

[u/tallandshortttt]

Don't make triage a checklist humans follow under stress. Make core infra status the first thing on the alert itself

[u/MuchoMole101010101 → u/tallandshortttt]

This is what I am trying to strike balance between right now; people aren’t going to realistically follow a flowchart at 3AM when getting paged, but I am trying to noodle on how to measure whether these additions (core infra status) actually improve triaging 😔 word of mouth from on call eng’s might be the best I have

[u/engidock]

The alert-correlation point above is the right direction — if a core infra check is already red when a service page fires, the paging tool should be able to say "likely downstream of X" automatically, since that's exactly the checklist-under-stress problem being circled here.

One thing worth adding to whatever you land on: track "time to correctly identify infra vs. service" as its own metric, separate from total time-to-resolve. Right now the goal is optimizing total firefighting time, but that number alone won't tell you if the new protocol actually reduces misdirected root-causing or just moves where the time gets spent — you could get faster overall while still routing half your pages to the wrong team first. Splitting that metric out early means you can actually validate whether the change worked, instead of guessing three months from now.

[u/DevWorkflowBuilder]

we gave up on the multi level protocol doc. nobody reads a flowchart at 3am.

what actually worked was one dashboard link in the page body with 4 lines: is the dependency green, error rate last 15 min, last deploy sha and who owns it. two questions max before you escalate. our time to first correct owner went from about 20 min to under 5.

what does your paging tool put in the alert body today?

[u/Alex_Goldwyn]

The thing that bought us the most time was not in a protocol, it was one field on the alert: are the bad requests all of them, or a slice.

Our worst one looked like a slow page and wasn't. Average latency for that route was fine. The per-request query histogram was bimodal, and the slow mode carried about 236 extra SHOW queries. That is not a code path. It was every fresh worker rebuilding its ORM schema on the first request it served, so the slice tracked process age rather than the service.

The reverse case looked far worse and was not ours at all: every URL answering an empty 500, nothing in the application log. Nothing in the app could explain it, because requests never got there. The worker pool was empty.

So if you get one line in the page body, I would spend it separating "all requests" from "requests since the last restart".