Kafka — Cheat Sheet

Tags: kafka, event-streaming, pub-sub, microservices, distributed-systems

Level: intermediate

TL;DR: Apache Kafka is a distributed event streaming platform used for high-throughput, low-latency data pipelines and decoupled microservice communication.

Key concepts

Broker: A single Kafka server node; multiple brokers collaborate to form a Kafka cluster.

Topic: A logical channel or category where producers publish events and consumers read them.

Partition: Topics are divided into partitions to scale across brokers and allow parallel processing.

Offset: A unique, sequential ID assigned to each message within a partition to track consumer reading position.

Producer: A client application that publishes (writes) events to one or more Kafka topics.

Consumer: A client application that subscribes to (reads) topics and processes the published events.

Consumer Group: A set of consumers cooperating to consume a topic; each partition is read by only one consumer per group.

KRaft / ZooKeeper: The consensus and metadata management layer (KRaft is the modern, ZooKeeper-less architecture).

Retention Policy: Kafka stores messages durably for a configured time or size limit, regardless of consumption status.

Typical questions and ready answers

Q: How does Kafka guarantee message ordering?

R: I always clarify that order is only guaranteed within a single partition, not across the entire topic. To ensure strict order for specific entities, I use a consistent message key so they route to the same partition.

Q: What happens if you have more consumers in a group than partitions?

R: The excess consumers will sit idle and do no work. I make sure to configure the number of partitions to be greater than or equal to the maximum number of concurrent consumers needed.

Q: How is Kafka different from traditional message queues like RabbitMQ?

R: I point out that Kafka is an append-only distributed log where messages are retained for a set period even after consumption, whereas RabbitMQ typically removes messages once they are acknowledged.

Pitfalls and common mistakes

Using acks=0 or acks=1 on the producer when data durability is critical, risking message loss during leader elections.

Designing topics with too few partitions initially, which limits future horizontal scalability of consumers.

Choosing a skewed message key, causing "hot partitions" where one broker handles most of the traffic while others are idle.

Failing to commit offsets properly, leading to duplicate message processing after a consumer restart or rebalance.

Snippets and commands

# Create a new topic with 3 partitions
kafka-topics.sh --create --topic user-signups --bootstrap-server localhost:9092 --partitions 3 --replication-factor 1

# Start a command-line producer
kafka-console-producer.sh --topic user-signups --bootstrap-server localhost:9092

# Start a command-line consumer reading from the beginning
kafka-console-consumer.sh --topic user-signups --from-beginning --bootstrap-server localhost:9092

# Check consumer group offsets and lag
kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-consumer-group

To review

[ ] Deep dive into KRaft consensus protocol and how it handles controller quorum.

[ ] Exactly-once processing semantics (EOS) and transactional producers.

[ ] Tuning consumer parameters like max.poll.records and session.timeout.ms to avoid rebalance loops.

Modifica direttamente o con Gemini Fai clic in un punto qualsiasi per digitare e modificare direttamente oppure seleziona il testo per chiedere a Gemini di apportare modifiche.