Incident Response — Cheat Sheet
Tags: incident-response, sre, devops, observability, postmortem, alerting
Level: intermediate
TL;DR: Structured workflow for detecting, triaging, mitigating, and reviewing production failures under pressure without causing secondary outages.
Key concepts
-
Incident response prioritizes service mitigation and customer impact reduction over root cause analysis during active outage.
-
P1 and P2 incidents require an incident commander to coordinate communications and delegate tasks.
-
Keep diagnosis separate from remediation to avoid making unverified system changes under pressure.
-
Establish dedicated war rooms and single-source status channels to reduce alert noise and communication overhead.
-
Observability pillars (metrics, logs, traces) must pinpoint blast radius before initiating rollback or traffic draining.
-
Automated rollback mechanisms and circuit breakers mitigate regressions faster than manual code hotfixes.
-
Post-incident reviews must remain strictly blameless to identify systemic process vulnerabilities and design flaws.
-
Action items from incident postmortems require clear engineering ownership and tracking in backlog sprint cycles.
Typical questions and ready answers
-
Q: How do you handle a critical production outage when the root cause is unknown?R: I first assign an Incident Commander and establish communication. Then I identify the blast radius and rollback recent deployments or apply mitigation like rate limiting before deep debugging.
-
Q: What is your approach to conducting a postmortem?R: I run a blameless timeline reconstruction focused on telemetry gaps, process failures, and architectural limits. Every identified root vulnerability gets converted into a prioritized backlog ticket with an owner.
-
Q: When do you decide to fail over or restart services during an outage?R: I trigger failover or restarts only after verifying the health of the downstream dependencies and confirming restart storms or cache stampedes won't amplify the outage.
Pitfalls and common mistakes
-
Debugging in production instead of rolling back to the last known healthy state.
-
Multiple engineers applying ad-hoc infrastructure changes concurrently without central coordination.
-
Communicating incident statuses across fragmented channels instead of a dedicated incident feed.
-
Restarting saturated database clusters simultaneously, triggering connection panics and thundering herds.
-
Treating human error as root cause rather than investigating why system design permitted the failure.
Snippets and commands
# Check system resource exhaustion under pressure
uptime && free -m && df -h
# Identify top resource-consuming processes
ps aux --sort=-%mem | head -n 10
ps aux --sort=-%cpu | head -n 10
# Check active network connections and sockets
ss -tulpn | grep LISTEN
ss -s
# View real-time container/pod logs during an ongoing incident
kubectl logs -f deployment/<app-name> --tail=100 --timestamps
kubectl rollout undo deployment/<app-name>To review
-
[ ] Practice rapid rollback playbooks for database schema migrations and stateful workloads
-
[ ] Review mean time to detect (MTTD) and mean time to resolve (MTTR) escalation matrix
-
[ ] Define standardized severity levels (P1 to P4) with precise SLA thresholds
-
[x] Configure health checks and automatic circuit breaker tripping on upstream service failure
References
-
Google Site Reliability Engineering (SRE) Handbook — Incident Management
-
PagerDuty Incident Response Documentation and Working Guides
-
The DevOps Handbook: How to Create World-Class Agility, Reliability, & Security in Technology Organizations
