Incident Management — Cheat Sheet

Tags: incident-management, sre, devops, observability, postmortem, on-call

Level: intermediate

TL;DR: Structured lifecycle for triaging, mitigating, and reviewing production failures quickly while maintaining clear stakeholder communication.

Key concepts

Severity levels (SEV-1 to SEV-4) classify outages by business impact, urgency, and customer blast radius.

The Incident Commander coordinates response efforts and delegates diagnostic tasks without debugging directly.

Primary response objective is rapid mitigation and service restoration rather than root cause analysis.

Status dashboards and timed broadcast updates keep external stakeholders informed during triage.

Dedicated communication channels isolate responders from organizational noise and unauthorized inputs.

Fast rollbacks, feature flag disables, and traffic rerouting serve as primary containment actions.

Blameless postmortems evaluate timeline accuracy, telemetry coverage, and architectural vulnerabilities.

Postmortem action items require owners and deadlines to prevent recurrence in engineering backlogs.

Runbooks define verified procedural steps for common failure modes and service failovers.

Typical questions and ready answers

Q: How do you handle an active Sev-1 production incident when the root cause is unknown?R: I prioritize immediate mitigation over live debugging by rolling back recent deployments, toggling off suspicious feature flags, or rerouting traffic.

Q: What is your role when operating as an Incident Commander?R: I coordinate responders, assign diagnostic streams to specific individuals, track timelines, publish status updates, and block external interruptions.

Q: How do you run an effective blameless postmortem after a major outage?R: I facilitate an objective review focused on telemetry gaps, brittle processes, and remediation work without attributing blame to individuals.

Pitfalls and common mistakes

Debugging root causes in production instead of restoring traffic via immediate mitigation.

Neglecting to designate a single Incident Commander, creating disjointed and duplicated engineering actions.

Failing to issue periodic updates to internal and external stakeholders during prolonged outages.

Holding postmortems that assign personal blame instead of identifying systemic technical and procedural flaws.

Leaving postmortem remediation tasks untracked and unprioritized in the engineering backlog.

Snippets and commands

# Triage failing pods across all namespaces
kubectl get pods -A --field-selector status.phase!=Running

# Rollback deployment to previous revision
kubectl rollout undo deployment/api-service -n production

# Verify rollout status
kubectl rollout status deployment/api-service -n production

# Inspect error spikes in reverse chronological order
journalctl -u api-service.service -p err -n 100 --no-pager

To review

[ ] Incident severity classification matrix and alerting threshold definitions

[ ] Automated failover procedures and database snapshot restore testing

[x] Blameless postmortem structure and timeline construction methods

[x] War room communication protocol and status page templates

References

Google SRE Book: Managing Incidents

PagerDuty Incident Response Guide

Atlassian Incident Management Handbook