Distributed systems · messaging
The two names every backend interview reaches for. They look similar (both move messages between services) but they are fundamentally different tools: Kafka is a durable log you read from, RabbitMQ is a broker that routes and deletes. Get that one distinction and most questions answer themselves. Companion to S6 · Distributed Patterns and S7 · Architectural Styles.
Kafka is a log; RabbitMQ is a queue. Kafka keeps every message on disk for a retention window and consumers track their own position (offset), so many consumers can read the same data and you can replay history. RabbitMQ pushes a message to a consumer and, once acknowledged, it is gone. "Do I need to re-read or replay this stream?" points to Kafka. "Do I need rich routing and a message to be handled once then forgotten?" points to RabbitMQ.
Fix these two pictures and the rest is detail. Almost every interview answer is a consequence of "log vs broker."
Messages are appended to a partition and kept for a retention period. Each consumer group remembers its offset (its bookmark). Reading does not delete. Many independent readers, replay from any point, huge throughput. Think event stream.
A producer sends to an exchange, which routes to one or more queues by rules. A consumer takes a message, acks it, and it is removed. Rich routing, per-message delivery, low latency. Think work queue and task dispatch.
Treating them as interchangeable "message queues." If you say "I'd use Kafka" for a job-dispatch problem that needs per-message retries and dead-lettering, or "I'd use RabbitMQ" to fan a firehose of events out to five analytics consumers that each replay history, the interviewer learns you do not understand the models. Name the model first, then the tool.
The comparison interviewers are really probing. Read every row as a consequence of log-vs-broker.
| Dimension | Kafka | RabbitMQ |
|---|---|---|
| Model | distributed append-only log | message broker with routing |
| Consumption | pull; consumer tracks offset | push; broker delivers, consumer acks |
| After read | message stays until retention expires | message deleted once acked |
| Replay history | yes, reset the offset | no, once gone it is gone |
| Ordering | guaranteed within a partition | per queue, weakens with redelivery / multiple consumers |
| Routing | simple: topic + partition by key | rich: direct, topic, fanout, headers |
| Throughput | very high (millions/s), sequential disk | high (tens of thousands/s), lower ceiling |
| Multiple consumers of same data | native: many consumer groups | needs a fanout exchange + a queue each |
| Per-message TTL / priority / DLQ | limited / manual | first-class |
| Scaling unit | partitions (add to parallelise) | queues + consumers (competing consumers) |
| Best at | event streaming, log of facts, analytics, replay | task queues, RPC, complex routing, per-message control |
If you can explain partition, consumer group, and offset clearly, you are most of the way there. The rest are follow-ups.
orders. Split into partitions.ISR = in-sync replicas. Survives broker loss."A topic is split into partitions; ordering is guaranteed only within a partition, set by the message key. Consumers in a group share the partitions for scale; a separate group re-reads the same stream independently. Each consumer commits an offset, so it can resume or replay." That single sentence answers half the Kafka questions.
The heart of RabbitMQ is routing: producers never talk to queues directly, they publish to an exchange that decides where it goes.
order.*.eu. The flexible workhorse."Producers publish to an exchange with a routing key; bindings decide which queues get it. direct matches exactly, topic matches patterns, fanout broadcasts. Consumers ack to delete, nack to requeue or dead-letter, and prefetch bounds how much one consumer holds for fairness and backpressure."
Straight from S6: both are at-least-once by default, so design for duplicates. "Exactly-once" is at-least-once delivery plus an idempotent consumer. Ordering is the other half interviewers probe.
| Guarantee | Kafka | RabbitMQ |
|---|---|---|
| At-least-once | default; commit offset after processing | default; ack after processing |
| At-most-once | commit offset before processing | auto-ack on delivery |
| Effectively exactly-once | idempotent producer + transactions, or idempotent consumer | idempotent consumer (dedupe on a key) |
| Ordering | strict within a partition | per queue, but lost once messages requeue or fan to many consumers |
| Ordering price | order needs same key, which can create a hot partition | strict order needs a single consumer, which caps throughput |
Do not promise exactly-once on the wire, it does not exist. Take at-least-once + an idempotent consumer: carry an idempotency key, check "seen this key?" before acting. Kafka adds idempotent producers and transactions for its own end-to-end path, but a dedupe-on-key consumer is the portable answer that works for both. This is the same move as the outbox and idempotency from S6.
Answer by the workload, not by taste. Start from "is this a stream of facts to read many ways, or tasks to dispatch once?"
flowchart LR
Q{"stream of EVENTS or queue of TASKS?"}
Q -->|"log of facts, many consumers read or replay"| K1["KAFKA
analytics, event sourcing, audit, CDC, metrics, tell everyone"]
Q -->|"high throughput, ordered per key, long retention"| K2["KAFKA"]
Q -->|"dispatch work once, then forget"| R1["RABBITMQ
background jobs, emails, image resize, RPC, request-reply"]
Q -->|"rich routing: topic, fanout, headers"| R2["RABBITMQ"]
Q -->|"per-message retries, priority, TTL, dead-letter"| R3["RABBITMQ"]
Q -->|"both shapes in one system"| B["USE BOTH
Kafka for the event backbone, Rabbit for task dispatch"]
classDef accent fill:#DCEBFE,stroke:#4A90D9,color:#09244B
classDef bad fill:#FEE4E2,stroke:#C8102E,color:#09244B
classDef good fill:#D6F5E3,stroke:#1B7F46,color:#09244B
class K1,K2 accent
class R1,R2,R3 bad
class B good
Kafka-like (log, replay, streaming): AWS Kinesis, Google Pub/Sub, Azure Event Hubs, Redpanda (Kafka-API compatible). RabbitMQ-like (queue, routing, per-message): AWS SQS (+ SNS for fanout), Google Cloud Tasks, Azure Service Bus. The log-vs-broker distinction carries across all of them.
The shapes that come up, and which tool fits each naturally.
Competing consumers (work queue)
N workers share a load, each message handled once. RabbitMQ one queue, many consumers, prefetch for fairness. Kafka one consumer group, partitions split across workers (parallelism capped by partition count).
Pub/sub (fan-out)
One event, many independent reactors. Kafka a consumer group per subscriber, all read the same log. RabbitMQ a fanout exchange to one queue per subscriber.
Event streaming / sourcing
The log of events is the source of truth; rebuild state by replay. Kafka is the natural home; retention + replay + compaction are built for it. RabbitMQ cannot replay.
Request / reply (RPC)
Send a task, get a result back on a reply queue with a correlation id. RabbitMQ is built for this. Doable in Kafka but awkward.
It is rarely either/or. A common shape: Kafka as the event backbone (every domain event from S8 lands here, many consumers, replayable), and RabbitMQ or SQS for task dispatch (send this one email, resize this one image, with retries and a dead-letter queue). Saying "I'd use both, here's the split" is a strong senior answer.
The ones that actually come up, and the one or two sentences that land them.
Q · Kafka vs RabbitMQ in one line?
Kafka is a durable log you read by offset and can replay; RabbitMQ is a broker that routes and deletes on ack. Streams of facts to Kafka, tasks to dispatch to RabbitMQ.
Q · How does Kafka scale and keep order?
A topic splits into partitions; consumers in a group share them. Order holds only within a partition, set by the message key. More partitions, more parallelism, but order is per key, not global.
Q · How do many services read the same Kafka events?
Each gives itself a different consumer group. Reads do not consume the data, so every group reads the full stream independently and at its own pace.
Q · How does RabbitMQ decide where a message goes?
Producers publish to an exchange with a routing key; bindings route to queues. direct exact-match, topic pattern, fanout broadcast.
Q · Do they guarantee exactly-once?
No. Both are at-least-once, so plan for duplicates. Get effectively-once with an idempotent consumer that dedupes on a key. Kafka also offers idempotent producers + transactions for its own path.
Q · A message keeps failing. What happens?
RabbitMQ nack it to a dead-letter exchange after N tries. Kafka has no per-message DLQ; you write failures to a separate "dead-letter topic" yourself and keep the offset moving.
Q · How do you handle backpressure?
RabbitMQ set prefetch so a consumer holds only N unacked messages. Kafka the consumer pulls at its own pace, lag (offset behind the log end) is the signal to add consumers or partitions.
Q · When would you use both?
Kafka as the event backbone (replayable domain events, many consumers), RabbitMQ or SQS for task dispatch (one-off jobs with retries and DLQ). Different shapes, different tools.
The mistakes that cost the point, and the habit that catches all of them.
The traps
The repeatable answer
1. name the shape: stream of facts, or tasks to dispatch?
2. pick the tool: log (Kafka) vs broker (RabbitMQ).
3. state the guarantee: at-least-once, so idempotent consumer.
4. name the ordering / routing need and its cost.
Say the shape before the tool. "This is a replayable stream read by several consumers, so Kafka, partitioned by tenant_id, at-least-once with a dedupe key" is a staff answer. "I'd use Kafka" is a coin flip.
Kafka is a log you replay; RabbitMQ is a queue you drain. Streams of facts and replay go to Kafka; routed, one-shot tasks with retries go to RabbitMQ; both are at-least-once, so make the consumer idempotent. Name the shape, then the tool.
Companion to S6 · Distributed Patterns and S7 · Architectural Styles. Further reading: Kafka docs · RabbitMQ tutorials · Kafka exactly-once / transactions · RabbitMQ dead-letter exchanges.