

During an Incident

Information on what to do during a major incident. See our severity level descriptions for what constitutes a major incident.

Security Incident?

If this is a security incident, you should follow the Security Incident Response process.

Don't Panic!

1. Join the incident chat (see information below).

   • Anyone is free to join the chat to observe and follow along with the incident.

2. Follow along with the chat, add any comments you feel are appropriate, but keep the discussion relevant to the problem at hand.

   • If you are not an SME, try to filter any discussion through the primary SME for your service. Too many people discussing at once get become overwhelming, so we try to maintain a hierarchical structure to the discussion if possible.

3. Follow instructions from the Incident Commander.

   • Is there no IC in the room?
     • Manually page them via Slack, with !ic page in Slack. XXX This will page the primary and backup IC's at the same time.
     • Never hesitate to page the IC. It's much better to have them and not need them than the other way around.

Steps for Incident Commander

Resolve the incident as quickly and as safely as possible, use the Deputy to assist you. Delegate any tasks to relevant experts at your discretion.

1. Announce in the #on-call room in Slack that you are the incident commander, and create a topic-specific Slack channel for the issue. For issues with associated JIRA tickets, the channel should be #tmp-ticket, e.g., #tmp-pit-123. For issues without JIRA tickets, the channel can be #tmp-pagerduty-number, e.g., #tmp-pd-1337. Post the name of the topic-specific channel in #on-call, and move conversation about the issue there.

2. Identify if there is an obvious cause to the incident (recent deployment, spike in traffic, etc.), and delegate investigation to specific people (not "can someone do this?").

   • When there are people with area-specific expertise in the channel, feel free to use them to assist in the analysis. They should be able to quickly provide confirmation of the cause, but not always. It's the call of the IC on how to proceed in cases where the cause is not positively known.

3. Identify investigation & repair actions (roll back, rate-limit services, etc) and delegate actions to specific on-call team members. Typically something like this (obviously not an exhaustive list), XXX SRE insight here

   • Bad Deployment: Roll it back.
   • Web Application Stuck/Crashed: Do a rolling restart.
   • Note that our runbooks for on-call are available on GitHub, and list many common types of alerts.

4. If the complexity of the incident requires voice/video communication, spin up a Zoom call, and paste the link into the Slack channel. Note, however, that decisions made in the Zoom call must be communicated in the Slack channel as well, to allow efficient collection of decisions for later analysis.

5. Once incident has recovered or is actively recovering, you can announce that the incident is over. This usually indicates there's no more productive work to be done for the incident right now.

   • Identify any post-incident clean-up work.
   • You may need to perform debriefing/analysis of the underlying root cause.

6. Once the call is over, you can start to follow the steps from After an Incident.

Steps for Subject Matter Experts

You are there to support the incident commander in identifying the cause of the incident, suggesting and evaluation repair actions, and following through on the repair actions.

1. Investigate the incident by analyzing any graphs or logs at your disposal. Announce all findings to the incident commander.

   • If you are unsure of the cause, that's fine, state that you are investigating and provide regular updates to the IC.

2. Announce all suggestions for resolution to the incident commander, it is their decision on how to proceed, do not follow any actions unless told to do so!

3. Follow instructions from the incident commander.

4. Once the call is over, you can start to follow the steps from After an Incident.