Monitoring IT — Cheat Sheet

Tags: monitoring, observability, metrics, prometheus, grafana, alertmanager, logging, tracing

Level: intermediate

TL;DR: End-to-end IT monitoring stack combining metrics, logs, and distributed traces to detect incidents and ensure system health.

Key concepts

Three pillars of observability: metrics, structured logs, and distributed tracing.

Metrics represent aggregated numerical values tracked over fixed time intervals.

Golden Signals of monitoring: latency, traffic, errors, and saturation.

USE method covers utilization, saturation, and errors for system resources.

RED method tracks rate, errors, and duration for microservices.

Push vs Pull models: Prometheus pulls metrics via HTTP endpoints; StatsD pushes metrics.

Alerting must trigger on symptoms impacting users rather than internal causes.

Service Level Indicators (SLIs) measure real-time compliance with Service Level Objectives (SLOs).

Log aggregation decouples storage and search from local application servers.

Distributed tracing propagates context IDs across service boundaries for bottleneck analysis.

Typical questions and ready answers

Q: How do you structure a production alerting strategy without causing fatigue?R: I route actionable alerts based on customer-facing SLO burn rates, grouping warnings into Slack and paging engineers only for active degradation.

Q: What is the difference between Prometheus pushgateway and standard scraping?R: Standard scraping pulls metrics periodically from long-lived targets, while Pushgateway buffers metrics from ephemeral batch jobs until Prometheus scrapes them.

Q: How do you differentiate between blackbox and whitebox monitoring?R: Whitebox inspects internal service internals via metrics and logs; blackbox tests external behavior from the outside like user-facing probes.

Pitfalls and common mistakes

Alerting on CPU utilization spikes instead of high error rates or request latency.

High-cardinality labels in Prometheus metrics causing memory exhaustion and crashes.

Unstructured text logging preventing fast filtering and indexing during critical outages.

Missing correlation IDs across microservice network boundaries during distributed calls.

Storing high-frequency metrics indefinitely without configuring downsampling or retention policies.

Snippets and commands

# prometheus.yml scrape job example
scrape_configs:
  - job_name: 'api-service'
    scrape_interval: 15s
    metrics_path: '/metrics'
    static_configs:
      - targets: ['localhost:8080']

Bash

# Verify system resource utilization directly via CLI
top -b -n 1 | head -n 15
journalctl -u app-service.service -n 50 --no-pager

To review

[ ] PromQL rate vs irate semantic differences

[ ] Distributed tracing W3C Trace Context propagation

[ ] Grafana alerting rules and routing policies

[x] Four Golden Signals definition and alerting logic

Jarvis Interviews
Jarvis