SLA KPI IT — Cheat Sheet

Tags: sla, slo, sli, kpi, devops, sre, itsm, observability

Level: intermediate

TL;DR: Operational framework to distinguish SLA, SLO, and SLI, track critical IT metrics (MTTR, MTBF, Error Budget), and answer interview scenarios on service reliability.

Key concepts

SLI (Service Level Indicator) is the direct quantitative measurement of service behavior (e.g., latency, error rate).

SLO (Service Level Objective) is the internal target goal set for an SLI (e.g., 99.9% successful requests over 30 days).

SLA (Service Level Agreement) is the external contractual commitment with clients, including business penalties upon breach.

Error Budget equals 100% minus SLO, defining acceptable failure and controlling release velocity vs. stability.

MTTR (Mean Time to Resolution) measures average duration required to recover from a failure.

MTTD (Mean Time to Detect) measures average time elapsed between incident occurrence and team awareness.

MTBF (Mean Time Between Failures) measures operational reliability between consecutive service breakdowns.

Availability tiers determine downtime allowances (e.g., 99.9% allows ~43.8 minutes downtime per month; 99.99% allows ~4.38 minutes).

Four Golden Signals for infrastructure monitoring: Latency, Traffic, Errors, and Saturation.

ITIL Incident Priority matrix determines SLA targets based on calculated Impact and Urgency levels.

Typical questions and ready answers

Q: How do you explain the difference between SLA, SLO, and SLI to non-technical stakeholders?R: I describe the SLI as the actual speedometer reading, the SLO as our agreed speed limit to run safely, and the SLA as the contractual fine imposed if we get caught speeding.

Q: What action do you take when a service burns through its Error Budget mid-quarter?R: I halt non-critical production feature deployments, redirect engineering focus to technical debt and reliability fixes, and tighten operational alert thresholds until stability recovers.

Q: How do you define meaningful SLAs for an internal microservice?R: I focus on caller experience by establishing SLOs on the Four Golden Signals, aligning downstream consumer expectations without introducing rigid legal SLA contracts internally.

Q: What is the difference between MTTR and MTBF, and which one do you optimize first in distributed systems?R: MTBF measures frequency of outages while MTTR measures recovery speed; in microservices, I prioritize reducing MTTR via automation and rollbacks because failures are inevitable.

Pitfalls and common mistakes

Setting 100% availability SLO targets, which is economically impractical and halts all deployment velocity.

Measuring SLIs from internal server metrics alone rather than synthesizing end-user client experience.

Agreeing to an external SLA equal to the internal SLO, leaving zero safety margin or remediation buffer.

Treating MTTR as a single metric without distinguishing detection time (MTTD) from actual repair time.

Over-alerting on non-actionable KPI dips instead of alerting strictly on burn rate against the SLO budget.

Snippets and commands

# Calculate monthly allowed downtime in minutes for target availability
# Usage: ./downtime.sh 99.9
awk "BEGIN { avail=$1; total_mins=30*24*60; downtime=total_mins*(1-(avail/100)); printf \"Allowed Downtime: %.2f mins/month\n\", downtime }"

YAML

# Prometheus alerting rule: High Error Budget burn rate over 1h window
groups:
  - name: latency-slo-alerts
    rules:
      - alert: HighBurnRateLatency
        expr: (sum(rate(http_requests_total{status=~"5.."}[1h])) / sum(rate(http_requests_total[1h]))) > (1 - 0.999) * 14.4
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Error Budget burning 14.4x faster than budget allocation"

To review

[ ] Multi-window multi-burn-rate alerting algorithms

[ ] SRE workbook calculation for composite SLOs across distributed dependencies

[x] Distinction between SLA, SLO, and SLI definitions

[x] Core formulas for MTTD, MTTR, MTBF, and availability percentages

References

Google Site Reliability Engineering: Implementing SLOs

The Site Reliability Workbook (O'Reilly)

ITIL 4 Service Operation & Incident Management Framework

Jarvis Interviews
Jarvis