Incident response SOP template

The worst time to design your incident process is during an incident. This template right-sizes the classic playbook for small teams: clear severity levels, one incident commander, honest customer updates on a clock, and a blameless review that actually changes something.

Detection and declaration

What counts as an incident

An incident is any unplanned event that degrades the service customers experience or puts data at risk — not every alert, and not only full outages. When in doubt between “incident” and “just a blip”, declare. Declaring is cheap; discovering at hour three that nobody owned it is not.

Severity levels

Keep three: S1 — service down or data at risk for many customers; all hands, customer comms mandatory. S2 — significant degradation or a single major customer down; incident process runs, comms as warranted. S3 — minor degradation with workaround; tracked, fixed in hours, no ceremony. Write one concrete example of each from your own history next to the definitions.

Declare loudly

Whoever spots it declares it: one message in the incident channel with severity guess, what’s observed, and a link to evidence. The declaration message starts the clock and the timeline — timestamp everything after it.

Roles

The incident commander

The first responder is the commander until handed off explicitly. The commander coordinates — assigns investigation, decides comms, calls severity changes — and does not personally debug once more than two people are involved. One commander at a time, named in the channel.

Everyone else

Responders investigate and report findings to the channel (not in DMs — the channel is the record). A comms owner is named for S1/S2 and owns the status page and customer updates so engineers aren’t drafting apologies mid-debug.

Customer communication

The update clock

S1: first public update within 15 minutes of declaration, then every 30 minutes even if the update is “still investigating”. S2: within an hour, then hourly. An honest “we know, we’re on it” beats silence by miles; customers forgive outages, not blackouts.

What an update says

Three sentences: what’s affected, what we know, when we’ll update next. Never speculate on cause; never promise a fix time you don’t have. The final update states the resolution plainly.

Resolution and the review

Stabilize, then fix

Prefer the fastest safe path to restore service — rollback, failover, feature-flag off — over diagnosing root cause live. Root cause is the review’s job; the incident’s job is making customers whole.

The blameless review

Within five working days, for every S1/S2: timeline, contributing causes (plural — there’s never just one), what shortened or lengthened the incident, and 1–3 concrete follow-ups with owners and dates. Blameless means the process failed, not the person. A review that produces no change was a meeting, not a review.

Quiz — pressure-test the basics

• A big customer reports errors but dashboards look normal. Declare or wait?
• Who talks to customers during an S1, and how often?
• You found the root cause 20 minutes in, but a rollback would restore service now. Which comes first?
• What must exist within five days of every S1?