SHTF! Call the e-team!

What your team needs from you before, during, and after a production fire

A production incident is the moment when the whole organization turns toward engineering and expects answers, direction, and resolution. As an engineering manager, you’re not the one writing the fix. Your job is to make sure the people who are writing it can actually do their job.

That requires being calm, having a process that already exists before anything breaks, and knowing exactly what you’re supposed to do while your engineers are elbow-deep in logs.

Before anything breaks

If you’re defining your incident response process during an incident, you’re already behind. The process needs to exist, be understood, and have been practiced before anything goes wrong.

The basics: who’s on call and how do they know it? What constitutes an incident versus a bug? This distinction matters more than it sounds: teams that can’t quickly categorize a problem waste time hesitating when they should be escalating. Clear severity definitions (P1 = production down, P2 = major degradation, P3 = noticeable issue without business impact) remove the hesitation. If the criteria are ambiguous, things stay quiet when they shouldn’t.

Define roles ahead of time. Incident lead who coordinates the technical response. Communications lead who handles stakeholder and customer updates. Technical responders who own the investigation. These roles don’t need to be the same people every time, but someone needs to know they’re in each role at the start of every incident.

Designate a single communication channel, one Slack or Teams channel where the team coordinates, distinct from wherever you send stakeholder updates. Mixing the two creates noise during the moment you need signal most.

Practice this before you need it. Walk through a past postmortem with the team. Run a tabletop exercise. Let junior engineers shadow an experienced responder during a lower-severity incident. The first time someone is in an incident response shouldn’t also be the first time they understand how it works.

Brief your cross-functional partners in calm periods. Product, leadership, and customer-facing teams should already understand what an incident looks like, how updates will flow to them, and what they should and shouldn’t do while engineering is responding. The more of this is established beforehand, the fewer distractions engineers face during the incident itself.

During the incident

Your job is to clear the path, not to be on it.

Engineers troubleshooting a production problem need focus. Every status request they have to handle, every leadership question they need to answer, every escalation that lands in the wrong channel is a context switch that slows the resolution. Your job is to absorb all of that.

Give leadership regular, brief updates: what’s the impact, what’s the current hypothesis, what’s being done, and what’s the current ETA. You don’t need to know the technical details to communicate this. “The team is investigating, currently looking at the database layer, no ETA yet” is more useful than silence.

Keep a running log of what decisions were made and when, where the investigation went, what was tried and ruled out. This is the raw material for the postmortem, and it’s only accurate if you capture it in real time.

Stay calm. This is the most unglamorous but most important thing. Engineers read their manager’s affect during a crisis. If you’re visibly panicked or sharp with people, that raises the team’s stress level at exactly the moment they need to think clearly. If you’re steady and organized, it signals that the situation is under control, even when it isn’t yet.

After the incident

The technical problem is resolved. Now the real work of an EM starts.

Run the postmortem within 48-72 hours, while memory is fresh. A good postmortem reconstructs what happened and why, using the timeline log from the incident. It identifies what made the failure possible, missing monitoring, an untested failure mode, a process gap, and produces action items that address those root causes. It does not name individuals as the cause. The blame-free culture discussion from another post applies here directly.

Acknowledge publicly what happened. If customers were impacted, own that clearly. Transparency after an incident builds more trust than a perfect track record would.

Recognize the people who responded. Name specifically what someone did, for example “pulled the right logs in the first five minutes and narrowed the problem scope significantly,” rather than a generic thank-you. Specific recognition lands differently.

Follow up on action items in the next sprint. A postmortem whose action items never ship is a wasted exercise. Make sure the highest-priority fixes make it onto the roadmap and get treated as real work, not good intentions.

Check in with the team in 1:1s after major incidents. Incidents are stressful, and not everyone processes stress at the same rate or in the same way. Some engineers feel energized by the response. Others feel shaken. Give them space to say how it landed for them, not just whether the system is healthy again.