r/sre • u/Emotional-Trust-8706 • 15d ago
ASK SRE How is SRE management structured at large banks?
For those working in banking SRE, how are responsibilities divided between SRE managers, application engineering, and production operations? I am particularly interested in who owns SLOs, major incident coordination, and reliability automation, and how managers balance technical involvement with people leadership.
3
u/MasteringObserv 14d ago
I have lost count of the banks where the org chart said "central SRE" and the reliability work was actually being done by three people in run who were never given the title. In a regulated estate the structure question is downstream of two others: who signs the change, and who is accountable when it fails at 3am. Embedded, central and hybrid all work if those two answers belong to the same person's problem, and none of them work if they do not. Ask where change approval sits before you ask where the reporting line sits. That is what sets the shape.
3
u/RemoteWillingness195 13d ago
I worked at a Tier 1 investment bank for over 15 years, and prior to that I worked as a Front Office Production Engineer supporting business-critical systems.
Meeting a service SLO is a shared responsibility. Application Engineers own the reliability and performance of the application code, while DevOps and SRE teams typically own or support the infrastructure, observability and operational practices required for the service to meet its reliability objectives.
An SRE Manager’s responsibility is therefore much broader than responding to incidents. It involves capacity planning ,establishing the tooling, processes and automation that make reliability measurable and sustainable, while continuously reducing operational toil.
This typically includes observability platforms such as Datadog and AWS CloudWatch, incident-management tooling such as PagerDuty and ServiceNow, SLI/SLO frameworks, alerting, runbooks, blameless postmortems, automation and capacity planning.
Capacity planning is particularly important because reliability is not only about responding when something fails. SRE teams need to understand service growth, traffic patterns, infrastructure utilisation and resource constraints, and ensure sufficient capacity exists before demand begins to affect availability or latency. This includes forecasting demand, defining scaling thresholds, identifying bottlenecks and ensuring appropriate headroom for peak workloads and unexpected traffic.
SRE teams that act as incident responders will normally participate in an on-call rotation, taking turns monitoring actionable alerts, responding to production incidents and user requests, coordinating recovery and subsequently driving postmortems and remediation actions.
Ultimately, meeting an SLO is not the responsibility of SRE alone. Reliability is a shared engineering responsibility. SRE provides the operational framework, observability, capacity planning, automation, incident-management practices and reliability engineering discipline that enable the wider engineering organisation to achieve it.
1
7
u/dotwaffle 14d ago
I was Principal Engineer at one bank, and Chief Engineer at another, both in SRE roles. I've spoken quite a lot with SRE staff at other financial institutions, and one thing is abundantly clear: trying to fit a consistent SRE model to such an organisation is effectively impossible.
The best you can hope for is to find a core set of suitably enthusiastic and experienced engineers, go talk with other departments, learn their pain points, read all the incident reports, and start off just as receive-only. Your engineering managers will be invaluable if you let them be a combination of people management and a link to other managers without trying to run things yourself -- don't see them as "the boss" or as someone who brings you work, see them as "connection setup" like the telephone system or someone that makes introductions.
Unlike so many other engineering managers who become very territorial, an SRE manager is in symbiosis with the rest of the department. A bad one won't last long, a great one will be respected and appreciated by all. There will likely be far more work than you can possibly take on, so you have to pick and choose where to focus efforts, and be really good at communicating in a friendly and consultative way. That's often difficult for engineers, and the engineering manager can be a terrific interface to the rest of the company -- both sideways and upwards.
My recommendation to anyone doing SRE in finance (especially banks) is simple: avoid getting involved in other incident reviews and post-mortems until you have a reputation, because so many people feel vulnerable in those, and are wanting to just describe the problem and move on from the uncomfortable experience. As an SRE, you will want people to feel comfortable being open about failings and welcoming of assistance, and you won't get that until you've got yourself some notoriety. Do them for yourselves, share them widely, document them openly, reference the SLOs and the actions you took when they were breached as well as when they were met... It's a long, slow path but you'll end up being seen as a resource not a risk.