Observability — Cheat Sheet

Tags: observability, monitoring, opentelemetry, tracing, metrics, logging, grafana, prometheus

Level: intermediate

TL;DR: Observability is measuring a system's internal state via external outputs using telemetry (logs, metrics, traces), focusing on unknown-unknown debugging rather than static checks.

Key concepts

Observability quantifies system internals based on telemetry outputs.

Monitoring detects known failure modes; observability explains novel failure states.

The three traditional telemetry pillars are logs, metrics, and distributed traces.

Metrics are aggregable numerical time-series measuring rate, errors, and duration.

The USE method tracks Utilization, Saturation, and Errors for hardware resources.

The RED method tracks Rate, Errors, and Duration for service request performance.

Structured logging uses JSON to preserve context, correlation IDs, and queryability.

Distributed tracing tracks a request lifecycle across microservices via trace and span IDs.

OpenTelemetry provides a vendor-neutral standard and SDK for instrumenting telemetry data.

High cardinality allows slicing metrics by granular attributes like tenant or user ID.

Typical questions and ready answers

Q: What is the difference between monitoring and observability?R: Monitoring tells me when a system is broken by tracking predefined thresholds. Observability lets me infer why it is broken by querying telemetry to debug unexpected states without deploying new code.

Q: How do you implement distributed tracing across microservices?R: I inject standard traceparent headers containing a unique trace ID and span ID into outbound HTTP or gRPC calls, extract them downstream, and emit spans via OpenTelemetry to a trace backend like Tempo or Jaeger.

Q: What are the Golden Signals in SRE?R: They are latency, traffic volume, error rate, and saturation. I use them as the primary dashboard to monitor service health and trigger alerts.

Q: When should you choose metrics over logs?R: I use metrics for real-time alerting, dashboards, and aggregations because they have fixed storage overhead. I use logs when I need raw payload details, stack traces, and granular event history.

Pitfalls and common mistakes

Generating excessively high cardinality tags in Prometheus, which exhausts memory.

Using unstructured text logs instead of structured key-value formats.

Logging high-volume events synchronously, which blocks request handling threads.

Missing context propagation headers, which breaks trace continuity between services.

Setting alerts on static infrastructure thresholds instead of user-impacting symptoms.

Sampling traces purely at head without tail-based sampling for errors and slow paths.

Snippets and commandsTypeScript

// OpenTelemetry manual span instrumentation in Node.js / TypeScript
import { trace, SpanStatusCode } from '@opentelemetry/api';

const tracer = trace.getTracer('payment-service', '1.0.0');

export async function processPayment(orderId: string, amount: number): Promise<void> {
  return tracer.startActiveSpan('processPayment', async (span) => {
    span.setAttribute('order.id', orderId);
    span.setAttribute('payment.amount', amount);

    try {
      await executeTransaction(orderId, amount);
      span.setStatus({ code: SpanStatusCode.OK });
    } catch (error) {
      span.recordException(error as Error);
      span.setStatus({
        code: SpanStatusCode.ERROR,
        message: (error as Error).message,
      });
      throw error;
    } finally {
      span.end();
    }
  });
}

Bash

# Verify OpenTelemetry Collector OTLP receiver endpoint via curl
curl -v -X POST "http://localhost:4318/v1/traces" \
  -H "Content-Type: application/json" \
  -d '{"resourceSpans":[]}'

To review

[ ] Context propagation mechanics in asynchronous message queues (Kafka headers)

[ ] Tail-based sampling configuration in OpenTelemetry Collector

[ ] PromQL rate vs irate functions for bursty metrics

[x] RED method versus USE method applications

[x] OpenTelemetry collector architecture and exporter pipelines

Jarvis Interviews
Jarvis