Prometheus Grafana — Cheat Sheet

Tags: prometheus, grafana, monitoring, metrics, observability, tsdb, dashboard

Level: intermediate

TL;DR: Prometheus scrapes and stores time-series metrics using PromQL; Grafana visualizes those metrics via customizable dashboards and alerts.

Key concepts

  • Prometheus: Pull-based TSDB (Time Series Database) that scrapes metrics from targets via HTTP.
  • Grafana: Visualization platform that connects to Prometheus (and other sources) to build dashboards.
  • PromQL: Prometheus Query Language, used to slice, dice, and aggregate time-series data.
  • Metrics types: Counter (only goes up), Gauge (can go up/down), Histogram (buckets of observations), Summary (quantiles).
  • Exporters: Agents (like Node Exporter) that expose metrics in Prometheus format for third-party systems.
  • Service Discovery: How Prometheus finds targets dynamically (e.g., Kubernetes endpoints, EC2 instances).
  • Alertmanager: Handles alerts sent by Prometheus, managing deduplication, grouping, and routing (e.g., to Slack/PagerDuty).
  • Pull vs Push: Prometheus *pulls* metrics; use Pushgateway for short-lived batch jobs that can't be scraped.

Typical questions and ready answers

  • Q: How do you handle high cardinality in Prometheus?

R: I drop unnecessary labels at scrape time using metric_relabel_configs, avoid putting unique IDs (like user IDs or raw IPs) in labels, and use recording rules to pre-aggregate heavy queries.

  • Q: Explain the difference between rate() and irate() in PromQL.

R: rate() calculates the per-second average over the entire time window, good for slow-moving or alerting metrics. irate() calculates the rate based on the last two data points, better for spiky graphs and zooming in.

  • Q: How does Grafana connect to Prometheus securely?

R: I configure Prometheus as a Data Source in Grafana, usually via internal network routing or using basic auth / TLS if crossing network boundaries, often provisioned via code (YAML/Terraform).

  • Q: What is a recording rule?

R: It's a way to pre-compute frequently needed or computationally expensive expressions and save the result as a new set of time series, speeding up Grafana dashboards.

Pitfalls and common mistakes

  • Storing high-cardinality data in labels (e.g., session IDs), which causes memory bloat and OOM crashes in Prometheus.
  • Using irate() for alerting rules; it's too sensitive to brief spikes and causes false positives (use rate() instead).
  • Forgetting that Counters reset on process restart; always use rate() or increase() which handle counter resets automatically.
  • Hardcoding thresholds in Grafana UI instead of managing alerts as code (via Prometheus rules or Grafana provisioning).
  • Scraping too often (e.g., 1s scrape interval) when the application only updates metrics every 15s.

Snippets and commands

# Example Prometheus scrape config for Node Exporter
scrape_configs:
  - job_name: 'node'
    scrape_interval: 15s
    static_configs:
      - targets: ['localhost:9100']

Snippet di codice

# Calculate HTTP 5xx error rate percentage
sum(rate(http_requests_total{status=~"5.."}[5m]))
/
sum(rate(http_requests_total[5m])) * 100

Snippet di codice

# Node exporter: CPU usage percentage
100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

To review

[ ] PromQL subqueries and offsets (e.g., [5m:1m], offset 1w).

[ ] Alertmanager inhibition rules vs routing tree.

[ ] Setting up Grafana provisioning (dashboards as JSON, datasources as YAML).

Jarvis Interviews
Jarvis